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MEMORANDUM  FOR  UNDER  SECRETARY  OF  DEFENSE  FOR  ACQUISITION 
ASSISTANT  SECRETARY  OF  THE  NAVY  (FINANCIAL 
MANAGEMENT) 

ASSISTANT  SECRETARY  OF  THE  AIR  FORCE 

(FINANCIAL  MANAGEMENT  AND  COMPTROLLER) 
INSPECTOR  GENERAL,  DEPARTMENT  OF  THE  ARMY 

SUBJECT:  Audit  Report  on  Duplication/Proliferation  of 

Weapon  Systems'  Modeling  and  Simulation  Efforts 
Within  DoD  (Report  No.  93-060) 


We  are  providing  this  final  report  for  your  information 
and  use.  The  report  notes  recent  positive  initiatives  to 
improve  the  management  of  models/simulations,  but  discusses 
deficiencies  requiring  further  action.  Comments  to  the 
draft  report  were  considered  in  preparing  this  final  report 
and  are  included  in  Part  IV,  Management  Comments. 

The  recommendations  are  subject  to  resolution  in 
accordance  with  DoD  Directive  7650.3,  in  the  event  of 
nonresponsive  comments.  We  request  that  the  Director  of 
Defense  Research  and  Engineering  reconsider  the  management 
position  on  Recommendation  B.I.,  regarding  the  need  for  an 
enforceable  single  verification,  validation,  and 
accreditation  standard.  Please  provide  those  final  comments 
by  May  1,  1993.  Provide  completion  dates  for  actions 
already  taken  and  the  estimated  dates  for  completion  of 
planned  actions  in  response  to  all  audit  recommendations. 

We  also  ask  that  your  comments  indicate  concurrence  or 
nonconcurrence  with  the  internal  control  weaknesses 
highlighted  in  Part  I.  No  further  response  from  the  other 
addressees  is  required. 

The  courtesies  extended  to  the  audit  staff  are 
appreciated.  If  you  have  any  questions  on  the  audit,  please 
contact  Mr.  Raymond  Spencer,  Program  Director,  at 
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(703)  614-3995  (DSN  224-3995)  or  Mr.  David  Vincent,  Project 
Manager,  at  (703)  693-0355  (DSN  223-0355).  Appendix  I  lists 
the  planned  distribution  of  this  report. 

Robert  J.  Lieberman 
Assistant  Inspector  General 
for  Auditing 

Enclosure 

cc: 

Secretary  of  the  Army 
Secretary  of  the  Navy 
Secretary  of  the  Air  Force 
Commandant  of  the  Marine  Corps 


Office  of  the  Inspector  General,  DoD 


AUDIT  REPORT  NO.  93-060  March  1,  1993 

(Project  No.  2AB-0016) 

DUPLICATION /PROLIFERATION  OF  WEAPONS  SYSTEMS ' 
MODELING  AND  SIMULATION  EFFORTS  WITHIN  DoD 

EXECUTIVE  SUMMARY 


Introduction.  Military  Departments  and  Defense  agencies 
have  a  hierarchy  of  models/ simulations  generally  oriented  to 
their  primary  mission.  In  recent  years  there  has  been  a 
significant  increase  in  the  number  and  frequency  of  use  of 
models/simulations  within  DoD.  Accordingly,  Congress  and 
DoD  management  have  become  concerned  about  the  potential  for 
duplication  and  proliferation  of  modeling  and  simulation. 

Objectives.  Our  objective  was  to  determine  if  redundant 
investment  is  being  made  by  DoD  for  modeling/simulation 
efforts  supporting  weapon  system  development.  In  addition, 
we  evaluated  internal  controls  used  to  prevent  unnecessary 
investment  in  models  and  simulations. 

Audit  Results.  In  June  1991,  the  Deputy  Secretary  of 
Defense  enacted  several  initiatives  directed  at  improving 
management  of  modeling  and  simulation  in  the  DoD. 
Specifically,  the  Defense  Modeling  and  Simulation  Office 
(DMSO)  was  established  as  a  focal  point  in  the  Department. 
Concurrent  with  our  audit  effort,  DMSO  drafted  a  Department 
of  Defense  Directive  that  addresses  certain  concerns 
discussed  in  this  report  and,  when  implemented,  should 
assist  in  improving  DoD  management  of  modeling  and  simula¬ 
tion  activities.  The  audit  did,  however,  disclose 
three  issues  requiring  further  management  attention. 

o  Model  and  simulation  projects  are  being  procured  and 
developed  within  the  DoD  without  adequate  coordination  and 
control.  The  DoD  has  not  established  policy,  procedures, 
guidance,  or  direction  to  manage  and  coordinate  Defense 
modeling  and  simulation  activities.  This  has  resulted  in 
redundant  models/simulations  and  a  proliferation  of  system 
architectures  and  libraries  (Finding  A) . 

o  The  vast  majority  of  models  and  simulations  current¬ 
ly  used  in  DoD  have  not  been  verified,  validated,  or 
accredited.  The  DoD  has  no  requirement,  criterion,  or 
standard  by  which  to  accomplish  a  verification,  validation, 
or  accreditation  process.  Weapon  system  requirement 


decisions,  development  decisions,  engineering  designs, 
operations,  and  test  and  evaluation  results  may  be  based  on 
computer-generated  data  that  is  inaccurate  or  misleading 

(Finding  B) . 

o  The  majority  of  models  and  simulations  used  in  DoD 
lack  adequate  configuration  management  and  documentation 
necessary  to  assure  ready  access  by  authorized  Defense 
personnel.  This  condition  exists  because  of  ineffective 
guidance  and  lack  of  oversight  by  DoD  Components  (Finding 
C)  . 

Internal  Controls.  There  is  a  lack  of  effective  internal 
controls  regarding  modeling/simulation  in  the  Military 
Departments  and  Defense  agencies.  Effective  policies, 
procedures,  and  guidelines  to  control  modeling/simulation 
development  and  proliferation  have  not  been  established. 
Additional  details  are  provided  in  Part  II  of  this  report. 

Potential  Benefits  of  Audit.  We  estimated  that  the  DoD 
could  avoid  as  much  as  $803  million  in  procurement,  military 
personnel,  and  operation  and  maintenance  costs  over  the 
6-year  Future  Years  Defense  Plan  by  reducing  duplication  and 
proliferation  of  models  and  simulations  by  establishing 
guidance  and  standards  for  development  and  by  strengthening 
internal  controls  (Appendix  G) . 

Summary  of  Recommendations.  We  recommended  that  the  Under 
Secretary  of  Defense  for  Acquisition  develop  policies  and 
responsibilities  related  to  investment,  internal  develop¬ 
ment,  interoperability  standards,  modification  of  existing 
assets,  and  maintenance  of  catalogues;  that  DoD  Components 
develop  policy,  guidance,  standards,  and  criteria  by  which 
verification,  validation,  and  accreditation  of  models  and 
simulations  is  to  be  accomplished;  and  the  Under  Secretary 
of  Defense  for  Acquisition  establish  a  DoD-wide 
configuration  management  and  documentation  policy  for  models 
and  simulations. 

Management  Comments.  The  Under  Secretary,  of  Defense  for 
Acquisition  concurred  with  all  recommendations.  The  Joint 
Staff  stated  that  the  subject  audit  provides  an  excellent 
assessment  of  the  status  of  DoD  policy  and  guidance  for 
modeling  and  simulation  standards  and  verification, 
validation,  and  accreditation.  The  Department  of  the  Army 
fundamentally  agreed  with  the  general  thrust  of  the  audit 
report.  The  Department  of  the  Navy  did  not  formally  respond 
but  informally  agreed  with  our  recommendations.  The  Naval 
War  College  concurred  with  all  recommendations  but  suggested 
that  the  threshold  reporting  criteria  in  Recommendation  A. 2. 
be  lowered.  The  Department  of  the  Air  Force  did  not 
respond. 
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PART  I  -  INTRODUCTION 


Background 

A  "model"  is  a  physical,  mathematical,  or  otherwise  logical 
representation  of  a  system,  entity,  phenomenon,  or  process. 
A  "simulation"  is  a  method  for  implementing  a  model  over 
time,  as  well  as  a  technique  for  testing,  analysis,  or 
training  in  which  real  world  and  conceptual  systems  are 
reproduced  by  a  model.  "Simulation"  is  also  defined  as  a 
model  of  a  "real  world"  situation.  The  terms  "model"  and 
"simulation"  are  often  used  interchangeably.  Design  of  a 
model  starts  with  assumptions  representing  this  "real  world" 
situation  as  a  mathematical  model  (equations) ,  list  of 
events,  or  a  combination  of  equations  and  events. 

In  DoD,  models  and  simulations  are  used  to  study  and  analyze 
various  scenarios  and  threats  in  a  combat  environment. 
Military  Departments  and  Defense  agencies  have  a  hierarchy 
of  models  for  this  purpose  that  are  generally  oriented  to 
their  primary  mission. 

Expanding  technological  capability  has  enabled  operations 
researchers  to  apply  computer-assisted  analysis  to  more 
diverse  fields  of  study  in  an  increasingly  sophisticated 
manner.  This  has  resulted  in  a  significant  increase  in  the 
number  and  frequency  of  use  of  modeling  within  the  DoD. 

Accordingly,  DoD  management  and  Congress  have  become 
concerned  about  the  potential  for  duplication  and 
proliferation  of  modeling  and  simulation.  In  June  1991,  the 
Deputy  Secretary  of  Defense  established  the  Defense  Modeling 
and  Simulation  Office  (DMSO) .  This  initiative  also  assigned 
responsibility  for  modeling  in  DoD  to  the  Under  Secretary  of 
Defense  for  Acquisition,  created  the  DoD  Executive  Council 
for  Models  and  Simulations  (EXCIMS) ,  and  addressed  the  need 
for  interoperability  standards  and  protocols. 

In  March  1992,  the  Joint  Staff  established  the  Joint 
Simulation  and  Interoperability  Division.  Objectives  for 
this  Joint  Staff  initiative  included  promoting  the 
application  of  modeling  and  simulation  in  joint  operational 
planning  and  execution,  education,  training,  exercises, 
operations  requirements,  joint  test  and  evaluation,  and 
doctrine  development  and  evaluation.  This  initiative  is  to 
promote  the  introduction  of  new  modeling  and  simulation 
technology  into  joint  operational  use  and  to  assist  the 
Force  Structure,  Resources,  and  Assessment  Directorate,  J-8, 
to  consolidate  the  modeling  and  simulation  plans  of  the 
commanders  in  chief,  cut  costs,  and  reduce  duplication. 


objectives 

Our  objective  was  to  determine  if  redundant  investment  is 
being  made  by  DoD  for  modeling  and  simulation  efforts 
supporting  weapon  system  development.  In  addition,  we 
evaluated  internal  controls  used  to  prevent  unnecessary 
investment  in  modeling. 

Scope 

The  audit  was  performed  from  December  1991  through 
August  1992.  We  visited  24  activities  and  evaluated 
62  models  and  simulations  randomly  selected  by  statistical 
sampling.  We  reviewed  and  analyzed  system  documentation; 
configuration  management  plans  and  controls;  and  evidence  of 
verification,  validation,  and  accreditation  for  each  model 
and  simulation  in  our  sample.  We  also  obtained  actual 
expenditures  for  the  most  recent  5-year  history  and  budget 
estimates  for  expenditures  for  the  upcoming  5 -year  period  at 
each  activity  visited.  In  addition,  we  evaluated  internal 
controls  and  organizational  relationships  related  to  this 
technical  area. 

This  economy,  efficiency,  and  program  audit  was  performed  in 
accordance  with  auditing  standards  issued  by  the  Comptroller 
General  of  the  United  States  as  implemented  by  the  Inspector 
General,  DoD,  and  accordingly  included  such  tests  of 
internal  controls  as  were  considered  necessary.  Appendix  H 
lists  activities  visited  or  contacted  during  the  audit. 

In  August  1990,  BDM/ITAC  Corporation,  under  contract  to  the 
Deputy  Director  for  Defense  Research  and  Engineering, 
submitted  to  DoD  a  compendium  of  models  and  simulations 
existing  within  the  Department.  This  BDM/ITAC  compendium 
was  not  represented  as  a  comprehensive  listing,  but  rather 
sought  to  establish  a  baseline  for  future  development  and 
management  of  modeling  and  simulation  to  support  Operational 
Test  and  Evaluation  (OT&E) . 

The  BDM/ITAC  compendium  identified  500  models  and 
simulations  categorized  into  13  mission  areas  defined  as 
either  a  generic  application  or  major  weapon  system 
application.  We  limited  our  audit  scope  to  the  341  models 
or  simulations  within  the  compendium  that  were  owned  or 
controlled  by  DoD.  Our  approach  used  this  population 
from  which  we  selected  62  models  for  analysis  and  review. 

Science  Applications  International  Corporation  (SAIC)  and 
the  Quantitative  Methods  Division  of  the  IG,  DoD,  provided 
technical  assistance  for  analysis  of  model  documentation; 
configuration  management;  and  verification,  validation,  and 
accreditation  (W&A) .  The  Quantitative  Methods  Division  of 
the  IG,  DoD,  also  assisted  in  sample  selection  and  in 
projecting  sample  results. 


2 


internal  Controls 


We  evaluated  the  effectiveness  of  internal  controls 
established  to  prevent  redundant  or  unnecessary  investment 
in  models  and  simulations.  The  audit  identified  material 
internal  control  weaknesses  as  defined  by  Public  Law  97-255, 
Office  of  Management  and  Budget  Circular  A-123,  and  DoD 
Directive  5010.38.  Controls  either  were  not  in  place  or 
were  generally  ineffective  at  most  activities  visited  during 
the  audit.  In  addition,  many  activities  visited  were  not  in 
compliance  with  applicable  DoD  regulations.  All  recommen¬ 
dations  cited  in  this  report,  if  implemented,  will  correct 
these  weaknesses.  A  copy  of  this  report  will  be  provided  to 
the  senior  official  responsible  for  internal  controls  within 
the  Office  of  the  Under  Secretary  of  Defense  for 
Acquisition. 

Prior  Audits  and  Other  Reviews 

The  General  Accounting  Office;  Inspector  General,  DoD;  and 
U.S.  Army  Audit  Agency  have  issued  four  reports  on  modeling 
and  simulation  from  December  1987  through  July  1991. 

U.S.  Army  Audit  Agency  Report  No.  91-A1.  "Development  of 
Computer-Based  Models  and  Simulations,11  July  1991.  The 
auditors  prepared  an  advisory  report  with  suggested  actions 
and  management  checklists  to  assist  managers  and  model  and 
simulation  developers.  The  Deputy  Under  Secretary  of  the 
Army  (Operations  Research)  endorsed  this  advisory  report 
stating,  "The  advisory  report  furnishes  a  sound  basis  for 
managing  model  and  simulation  development  by  all  levels  of 
management."  The  report  concluded  that  policies  and 
procedures  were  needed  to  ensure  that  models  and  simulations 
were  properly  managed,  justified,  approved,  and  controlled. 
Specific  recommended  actions  include  requiring  new  model  and 
simulation  development  efforts  to  conform  to  DoD  and 
Department  of  the  Army  documentation  standards,  requiring 
proper  verification  and  validation  efforts  be  documented 
before  new  models  and  simulations  are  accredited  for  use  in 
Army  applications,  and  requiring  revalidation  of  models  and 
simulations  each  time  changes  are  made. 

Appendix  F  summarizes  other  reports  issued  on  modeling  and 
simulation. 

Other  Matters  of  Interest 

Numerous  inquiries  and  studies  have  been  conducted  . on 
computer— assisted  representations  of  warfare  in  DoD  which 
include: 

o  Defense  Science  Board  Task  Force  on  Computer 
Applications  to  Training  and  Wargaming  (1988), 
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o 


Defense  Science  Board  Task  Force  on  Improving  Test 
and  Evaluation  Effectiveness  (1989),  and 

o  DoD  Simulation  Policy  Study  (1990). 

All  three  studies  concluded  that  there  were  managerial  and 
technical  deficiencies  in  implementing  computer  technology 
to  the  analysis  of  warfare  in  the  DoD.  Specific  managerial 
deficiencies  included  modeling  and  simulation  users  and 
developers  lacking  a  broad  DoD  perspective,  Services  and 
commands  pursuing  courses  of  action  that  often  led  to  a  lack 
of  interoperability  between  simulations,  and  a  lack  of 
configuration  management. 

Technical  problems  included  the  need  to  improve  W&A 
processes  and  procedures;  the  need  for  common  architectures, 
standards,  and  interoperability;  and  the  need  for  research 
and  development. 

In  a  report  entitled  "A  Review  of  Study  Panel 
Recommendations  for  Defense  Modeling  and  Simulation,"  June 
1992,  and  completed  for  the  DMSO,  the  Institute  for  Defense 
Analyses  summarized  179  recommendations  made  by  senior 
study  panels  over  a  16-year  period  concerning  Defense 
modeling  and  simulation.  Certain  common  themes  permeate  all 
of  these  recommendations. 

Specifically  the  reports  recommended  that  the  responsibility 
for  modeling  and  simulation  should  be  raised  to  senior 
levels  of  the  Services  and  Joint  Staff;  Defense  management 
should  provide  policy  to  guide  greater  cooperation  and 
information  sharing  on  development,  implementation, 
maintenance,  verification,  validation,  and  accreditation  of 
Defense  models  and  simulations;  and  the  need  for  a 
broadened  perspective  as  to  how  and  where  modeling  and 
simulation  technology  should  be  applied.  Other  common 
themes  included  the  need  for  systematic  planning,  improving 
technology,  improving  technical  quality,  and  increasing 
applications  in  system  acquisition  and  test  and  evaluation. 

Another  matter  of  interest  is  Army  Regulation  5-11,  "Army 
Model  and  Simulation  Management  Program,"  July  10,  1992. 
This  regulation  establishes  the  Army  Model  and  Simulation 
Management  Program  (AMSMP)  and  prescribes  policies  and 
responsibilities  for  the  management  of  modeling  and 
simulation  used  for  all  purposes  within  the  Army.  It  also 
provides  a  management  structure  and  regulatory  guidance 
governing  the  development,  acquisition,  and  use  of  modeling 
and  simulation  in  specific  functional  disciplines.  These 
functional  disciplines  include  research  and  development, 
test  and  evaluation,  education  and  training,  production  and 
logistics,  and  analysis. 
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The  AMSMP  establishes  the  Army  Model  and  Simulation 
Executive  Council  and  the  Army  Model  and  Simulation 
Management  Office.  The  Army  Model  and  Simulation  Executive 
Council  is  to  act  as  a  study  advisory  group  and  to  recommend 
policy  guidance  to  the  Deputy  Under  Secretary  of  the  Army 
(Operations  Research)  concerning  the  management  of  all  Army 
modeling  and  simulation. 

The  Army  Model  and  Simulation  Management  Office  (AMSMO)  is 
to  act  as  executive  secretariat  for  the  Army  Model  and 
Simulation  Executive  Council,  establish  and  maintain  a 
catalogue  of  Army  models  and  simulations,  and  act  as  focal 
point  for  Army  matters  in  dealing  with  the  DMSO. 

Army  Regulation  5-11  specifically  requires: 

o  Development  of  an  Army  Model  Improvement  Master 
Plan,  which  is  to  include  specific  management  procedures  and 
processes  for  administration  of  the  Army  Model  Improvement 
Program. 

o  Assignment  of  responsibility  to  the  AMSMO  to  develop 
and  maintain  an  automated  Model  and  Simulation  Master 
Catalogue.  In  addition,  it  establishes  the  AMSMO  as  a 
single  point— of— contact  for  input  into  Defense  cataloguing 
systems . 

o  Verification,  validation,  and  accreditation  of  all 
models  and  simulations  developed  after  the  effective  date  of 
the  regulation. 

o  Configuration  management  to  be  applied  to  any  model 
or  simulation  developed  wholly  or  partly  with  Army  funding. 

o  Use  of  standardized,  non-redundant  data  structures 
for  input  and  output  data  for  models  and  simulations 
developed  after  the  effective  date  of  the  regulation. 
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PART  II  -  FINDINGS  AND  RECOMMENDATIONS 


A.  DUPLICATION .  REDUNDANCY.  AND  PROLIFERATION 

Model  and  simulation  projects  are  being  procured  and 
developed  within  the  DoD  without  adequate  coordination  and 
control.  This  situation  is  occurring  because  DoD  has  not 
established  policy,  procedures,  or  guidance  to  manage  and 
coordinate  Defense  modeling  and  simulation  activities.  This 
situation  has  resulted  in  redundant  models  and  simulations 
and  proliferation  of  system  architectures  and  libraries. 
During  this  audit,  we  estimated  that  only  a  10-percent 
reduction  in  the  number  of  redundant  models  and  simulations 
represents  as  much  as  $803  million  of  unjustified  funds  over 
the  6-year  Future  Years  Defense  Plan. 

DISCUSSION  OF  DETAILS 


Background 

When  the  Deputy  Secretary  of  Defense  approved  creation  of 
the  DMSO  on  June  21,  1991,  he  stated  that  the  new  office  was 
intended  to  support  the  Undersecretary  of  Defense  for 
Acquisition  "in  strengthening  the  use  of  modeling  and 
simulation  in  joint  education  and  training,  research  and 
development,  test  and  evaluation,  and  operations  and  cost 
analysis. " 

Properly  applied,  modeling  and  simulation  can  generate 
useful  information  quickly  and  relatively  inexpensively  when 
compared  to  the  cost  of  conducting  actual  field  exercises 
and  multiple  live  fire  tests  of  major  weapons  systems. 
Modeling  can  be  used  to  represent  scenarios,  threats,  direct 
engagement  counter fire,  electronic  warfare  interaction,  and 
a  combat  environment  on  a  scale  impractical  in  the  "real 
world."  Modeling  can  also  be  used  in  the  conceptual  design 
of  a  developing  weapon  system  as  a  simple  input-output  black 
box  or  as  a  functional  representation  of  the  weapon  system. 

Different  funding  sources  make  it  difficult  to  identify 
precisely  the  resources  within  DoD  dedicated  to  modeling  and 
simulation.  In  1991  the  Deputy  Director  of  Defense  Research 
and  Engineering  (Plans  and  Resources)  estimated  that  DoD 
would  spend  as  much  as  $939  million  in  FY  1992  on  modeling 
and  simulation  supporting  weapon  system  development. 

As  part  of  our  audit,  we  attempted  to  verify  this  estimate. 
At  24  activities  visited,  we  identified  and  collected  data 
on  FY  1992  expenditures  directly  related  to  the  development 
and  use  of  computer  modeling.  Using  this  data,  we 
estimated  that  these  24  activities  will  spend  as  much  as 
$198  million  on  modeling  and  simulation  for  FY  1992. 
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Based  on  cost  data  from  these  activities  and  extrapolating 
from  the  BDM/ITAC  Corporation  sample  data,  we  estimate  that 
during  FY  1992,  DoD  expenditures  for  modeling  and  simulation 
supporting  weapon  system  development  could  be  as  much  as 
$1.3  billion  to  $1.6  billion  (Appendix  A). 


DoD  Management  of  Modeling  and  Simulation  Activities 

Within  DoD  much  work  on  various  projects  related  to  modeling 
and  simulation  has  generated  a  multitude  of  computer  codes, 
with  different  designs  and  language  implementation,  but 
frequently  oriented  toward  the  same  goal.  Technological 
advances  now  allow  users  to  solve  problems  and  share 
information  to  an  extent  not  formerly  possible. 

Historically,  computer  models  have  been  developed  within 
Components  to  address  specific  mission-related  issues  and 
questions.  Only  in  the  last  5  years  has  the  need  for  a  DoD 
perspective  regarding  the  management  of  modeling  and 
simulation  been  recognized. 

DoD  Components  are  aware  of  the  need  for  modeling  and 
simulation  standards  and  protocols  and  are  attempting  to 
solve  this  problem  at  the  Component  level.  The  lack  of 
guidance  from  the  Office  of  the  Secretary  of  Defense  (OSD) 
level  has  resulted  in  standardization  efforts  that  are  not 
coordinated  and  controlled. 

During  our  review,  we  were  unable  to  identify  any  formal  OSD 
level  direction,  guidance,  or  policy  statements  regarding 
management  of  or  investment  in  computer  modeling.  Guidance 
for  the  consistent  development  of  modeling  and  simulation 
plans  by  DoD  Components  has  not  been  furnished.  Two  posi¬ 
tive  steps  we  did  note  were  the  issuance  of  the  Deputy 
Secretary  of  Defense  memorandum,  dated  June  21,  1991, 
"Modeling  and  Simulation  Management  Plan"  and  the  Under 
Secretary  of  Defense  for  Acquisition  memorandum  dated 
July  22,  1991,  which  established  the  Defense  Modeling  and 
Simulation  Office. 

Subsequent  to  establishment  of  the  Defense  Modeling  and 
Simulation  Office,  two  draft  initiatives  have  been  issued 
that  would  provide  these  requisite  DoD  policies  and  assign 
responsibilities  for  the  management  of  these  endeavors. 
Specifically,  Draft  DoD  Directive  50XX.XX,  "Modeling  and 
Simulation,"  August  1992,  if  approved,  would  establish  DoD 
policies  and  assign  roles  and  responsibilities  for  the 
management  of  modeling  and  simulation.  The  USD (A)  would  be 
responsible  for  strengthening  the  use  of  modeling  and 
simulation  within  DoD  and  for  issuing  plans,  policies, 
directives,  procedures,  and  publications  for  management  and 
advancing  modeling  and  simulation. 
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In  March  1992,  the  Joint  Staff  prepared  a  draft  of  Chairman, 
Joint  Chiefs  of  Staff,  Memorandum  of  Policy  No.  68, 
"Management  of  Joint  Modeling  and  Simulation."  If  approved, 
this  memorandum  would  establish  policy  and  procedures  and 
assign  responsibilities  for  the  management  of  joint  modeling 
and  simulation  activities  and  would  create  a  Joint  Modeling 
and  Simulations  Panel. 

Internal  Controls  Related  to  Modeling  and  Simulation 
Investments 

Internal  management  controls  are  those  policies,  procedures, 
and  practices  established  to  ensure  that  DoD  Components 
manage  resources  effectively  and  efficiently.  Effective 
internal  controls  that  would  prevent  or  minimize  duplication 
and  proliferation  in  modeling  and  simulation  activities 
would  include  comprehensive  inventories,  standardized 
procedures  for  approving  new  development  or  modifications, 
and  accounting  procedures  to  track  costs. 

Audit  Sample.  At  the  24  locations  visited,  we 
evaluated  existing  internal  management  controls  of  modeling 
activities.  We  also  reviewed  policies  and  procedures  for 
approving  development  of  new  models  and  simulations  and  the 
modifications  made  to  existing  models.  In  addition,  we 
reviewed  inventories,  lists,  compendiums,  or  catalogues  and 
examined  available  cost  information. 

Policies  and  Procedures  for  New  Development - or 

Modifications.  We  found  that  policies  and  procedures  for 
approving  acguisition  or  development  of  a  new  model  or 
modification  to  an  existing  model  vary  extensively 
throughout  DoD.  Models  developed  or  modified  "in-house"  are 
subjected  to  a  rigorous  review  and  approval  process  at  some 
locations  but  have  virtually  no  formal  review  or  approval 
process  at  others.  This  variation  is  evident  even  when  the 
activities  are  in  the  same  Service.  Specific  examples  of 
policies  and  procedures  in  place  at  three  Department  of  the 
Air  Force  activities  are  as  follows: 

o  The  Air  Force  Wargaming  Center,  Maxwell  Air  Force 
Base,  Alabama,  has  a  formal,  documented  approval  process  for 
in— house  development  of  models  and  simulations.  .  Written 
requests  for  development  or  modification  are  submitted  to 
the  Systems  Directorate.  The  Directorate  Chief  has  approval 
authority  for  projects  estimated  not  to  exceed  2  staff-years 
effort.  Projects  that  exceed  this  threshold  are  forwarded 
to  Air  University  Command  for  approval. 

o  The  Air  Force  Electronic  Warfare  Center,  San 
Antonio,  Texas,  has  a  less  formal  approval  process.  A 
director  assigned  a  project  or  study  will  determine  if 
development  or  modification  of  an  existing  model  or 
simulation  is  necessary.  If  so,  the  director  then 
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determines  if  this  development  or  modification  can  be  done 
in-house.  If  the  director  decides  to  do  the  work  in-house, 
the  necessary  engineers,  programmers  or  analysts  will  be 
tasked  to  complete  the  project. 

o  The  Air  Force  Studies  and  Analysis  Agency,  Pentagon, 
Washington,  D.C.,  has  no  formal  policies  or  procedures  for 
approving  or  tracking  in-house  modifications.  Analysts 
performing  the  study  or  analysis  make  whatever  changes  they 
deem  necessary  to  complete  their  assignment. 

Inventories  of  Models  and  simulations.  The  likelihood 
that  unnecessary  development  of  new  computer  models  will 
take  place  is  increased  if  DoD,  the  Military  Departments, 
and  Defense  agencies  do  not  maintain  an  accurate  inventory 
of  existing  model  and  simulation  assets  and  their  respectiye 
capabilities.  Several  activities  visited  during  our  audit 
were  unable  to  provide  us  with  an  accurate  inventory. 

The  U.S.  Army  Missile  Command  (MICOM) ,  Huntsville,  Alabama, 
provided  the  audit  team  with  a  list  of  models  that  did  not 
include  any  of  the  five  models  on  the  BDM/ITAC  compendium, 
from  which  our  sample  was  drawn.  However,  subsequent 
discussion  with  MICOM  personnel  disclosed  that  the  Command 
either  had  been  or  was  currently  using  all  five  of  these 
models.  A  MICOM  official  stated  that  because  of  the 
decentralized  management  structure,  he  doubted  if  anyone 
knew  how  many  models  were  in  use  at  the  activity. 

Similarly,  The  U.S.  Army  Aviation  and  Troop  Command,  Saint 
Louis,  Missouri,  initially  reported  only  three  computer 
models  in  its  inventory.  Subsequent  discussions  with 
various  Command  sources  resulted  in  the  audit  team  being 
given  a  list  of  more  than  100  models  and  simulations 
currently  in  use  at  the  Command. 

Modeling  and  simulation  costs.  One  of  the  first  steps 
in  effectively  managing  expenditures  is  to  accurately 
identify  both  current  and  prior  spending.  At  22  of  the 
activities  visited,  the  cost  of  a  model  or  simulation 
developed  by  a  Government  contractor  was  readily 
identifiable.  However,  the  Defense  Advanced  Research 
Projects  Agency  and  the  Army  Strategic  Defense  Command 
contended  that  costs  associated  with  modeling  and  simulation 
were  so  embedded  in  various  analyses,  studies,  or 
development  contracts  that  it  would  be  nearly  impossible  to 
extract  the  costs  incurred  for  the  portions  involved  in 
modeling. 

Duplication  and  Proliferation  of  Models  and  Simulations 

The  completed  BDM/ITAC  compendium  was  provided  to  us<  as 
being  the  most  comprehensive  list  of  models  and  simulations 
supporting  weapon  system  development  within  DoD.  We 
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selected  our  statistical  sample  from  the  BDM/ITAC 
compendium,  which  listed  500  models  and  simulations, 
identified  by  mission  issue  area.  Of  these  500  models  and 
simulations,  159  were  owned  or  access-controlled  by  non-DoD 
interests.  From  the  remaining  341  models  owned  or 
controlled  by  DoD,  we  randomly  selected  our  statistical 
sample  of  62  models. 

The  BDM/ITAC  compendium  categorized  these  models  and 
simulations  into  "mission  issue  areas"  whose  dimensions  or 
functional  characteristics  could  involve  common  algorithms 
or  data  or  both.  Many  of  these  computer  models  had 
functional  characteristics  that  were  defined  by  two  or  more 
"mission  issue  areas."  Specifically,  of  the  341  models 
owned  or  controlled  by  DoD: 

o  87  (26  percent)  had  a  Space  component, 

o  86  (25  percent)  had  an  Electronic  Warfare  component, 

o  79  (23  percent)  had  a  C3I  component, 

o  67  (20  percent)  had  an  Air-to-Air  component, 

o  43  (13  percent)  had  a  Force-on-Force  scenario, 

o  42  (12  percent)  had  a  Global  scenario,  and 

o  40  (12  percent)  had  an  Air-Land  component. 

During  the  audit  we  continued  to  collect  lists,  compendiums, 
catalogues,  and  libraries  with  additional  computer  models 
related  to  weapon  system  development.  Eliminating 

duplications  on  lists  obtained  from  different  sources,  we 
identified  an  additional  1,400  unique  models  or  simulations 
available  for  direct  or  indirect  support  of  weapon  system 
development.  DoD  and  its  Components  owned  or  controlled  an 
additional  1,208  of  these  models  or  simulations  beyond  those 
identified  in  the  BDM/ITAC  compendium. 

Using  criteria  set  in  a  list  of  Technical  Questions 
(Appendix  B) ,  the  audit  team's  technical  experts  identified 
duplicate  and  redundant  models.  These  models  were  developed 
to  analyze  mission  effectiveness,  electronic  warfare  mission 
planning,  aircraft  penetration,  deployment  of  communications 
equipment,  transportation  and  logistics,  and  strategic 
defense  (Appendix  C)  .  As  with  the  "mission  issue  areas**  of 
the  BDM/ITAC  compendium,  many  models  and  simulations 
included  in  our  sample  had  characteristics  that  could  be 
categorized  into  two  or  more  functional  areas.  Specifi¬ 
cally,  of  the  62  models  and  simulations  sampled: 

o  34  (55  percent)  had  measures  of  system 
effectiveness, 

o  26  (42  percent)  had  a  Many-on-Many  scenario, 
o  25  (40  percent)  had  measures  of  system  performance, 
o  24  (39  percent)  had  an  Air-to-Air  scenario,  and 
o  21  (34  percent)  had  Electronic  Warfare/ Electronic 
Countermeasures  components . 


These  suggest  the  possibility  of  overlap  in  function  or  data 
and  represent  duplication  and  proliferation  in  development 
of  these  computer  models. 

The  BDM/ITAC  compendium  was  intended  to  be  a  representative 
inventory  of  models  and  simulations  available  to  DoD  in 
Fiscal  Year  1988  for  Operational  Test  and  Evaluation 
purposes.  By  March  1992,  only  14  of  the  62  models 
originally  selected  from  the  compendium  in  our  random  sample 
were  still  in  active  use.  Assuming  this  same  ratio  of 
active  to  inactive  computer  models  for  the  entire  BDM/ITAC 
compendium  and  assuming  this  can  be  extended  to  the 
compendium  prepared  by  the  audit  team,  any  given  model  or 
simulation  has  a  relatively  short  "useful  life." 

This  has  two  implications.  First,  significant  effort  is 
devoted  to  developing  new  "stand-alone"  models  throughout 
DoD.  Second,  since  little  effort  is  directed  at  reusing  all 
or  parts  of  existing  models,  prior  investment  is  essentially 
lost.  Constantly  developing  new  models  without  attempting 
to  reuse  existing  ones  results  in  costly  duplication  of 
effort. 

Recently  a  joint  program  was  established  to  develop  a 
standard  modeling  architecture  and  simulation  support  system 
for  development  and  analysis  of  validated  digital  threat 
models.  The  Joint  Modeling  and  Simulation  System  (J-MASS) 
program  has  efforts  underway  to  use  parts  of  existing  models 
and  simulations  as  appropriate  and  to  develop  reusable  code 
and  data. 

Proliferation  of  System  Architectures 

The  term  "architecture"  in  the  context  of  modeling  and 
simulation  refers  to  information  formats  (syntax) , 
information  content  (semantics) ,  and  physical  connections 
which  join  one  model  or  simulation  to  another.  Any 
architectural  relationship  must  have  at  least  the  first 
two  attributes,  i.e.,  some  information  format  and  content. 
For  purposes  of  our  audit,  we  are  defining  "architecture"  as 
a  collection  of  interface  standards,  a  common  language,  and 
a  conceptual  framework  for  modeling  and  simulation  issues. 

The  broad  and  diverse  uses  of  models  and  simulations  within 
DoD  reguire  that  overall  architectural  standards  be 
established  to  allow  interoperability  of  models  and 
simulations,  thereby  increasing  the  sharing  of  resources 
among  DoD  Components. 

We  identified  a  number  of  system  architectures  in  various 
development  stages.  Most  of  these  architectures  are  widely 
accepted  within  their  respective  user  communities.  However, 
at  the  DoD  level,  these  projects  are  not  visible.  Because 
there  are  no  interface  standards,  we  feel  these  system 
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architectures  under  development  are  unlikely  to  be 
interoperable  with  other  architectures  or  systems. 
Accordingly,  because  there  are  no  interface  standards,  this 
ongoing  investment  in  development  of  system  architectures 
probably  will  not  be  productive  since  it  will  not  result  in 
interoperability. 


Modeling  and  Simulation  Libraries 

We  observed  an  unnecessary  proliferation  of  modeling  and 
simulation  catalogues  throughout  DoD.  Many  DoD  Components 
are  commissioning  or  publishing  catalogues  and.  lists  of 
models  and  simulations  that  are  limited  in  scope. 
Information  contained  in  these  catalogues  varies  greatly 
because  there  is  no  DoD  standard  or  guidance  as  to  what 
information  should  be  included. 

The  most  comprehensive  list  of  models  we  identified  was  the 
"Catalog  of  Wargaming  and  Military  Simulation  Models,"  12th 
Edition,  published  by  the  Joint  Staff,  Force  Structure, 
Resource  and  Assessment  Directorate  (J-8) .  This  list 
contains  522  distinct  models  and  simulations. 

During  our  audit,  we  identified  more  than  1,900  distinct 
models  and  simulations  listed  in  30  individual  catalogues  or 
listings.  Our  analysis  indicates  that  more  than  700  were 
listed  in  multiple  catalogues.  We  also  found  that  many  DoD 
Components  were  developing  and  using  models  or  simulations 
not  listed  in  any  catalogue  or  reference  source. 

Absence  of  a  central  library  resource,  standardized 
listings,  and  a  mandatory  reporting  requirement  contributes 
to  redundant  investment  because  users  have  no  reliable  way 
to  determine  what  models  or  simulations  already  exist  that 
might  satisfy  their  immediate  requirement. 

DMSO  has  initiated  the  Modeling  and  Simulation  Information 
System  as  a  prototype  electronic  bulletin  board  and 
repository.  The  Modeling  and  Simulation  Information  System 
will  have  clearinghouse  capabilities  to  include;  a  directory 
of  modeling  and  simulation  catalogues,  and  lists  of 
significant  events  and  documents.  Full  operational 
capability  is  expected  to  be  achieved  in  June  1993. 

Potential  Monetary  Benefits 

At  24  activities  visited,  we  identified  and  collected  data 
on  actual  and  budgeted  expenditures  for  FY  1992  directly 
related  to  modeling  and  simulation  activities.  This  cost 
data  included  funding  for  operations  and  maintenance, 
military  personnel,  procurement,  and  research,  development, 
test  and  evaluation.  We  then  summarized  the  funding  data  to 
determine  an  aggregate  cost  for  modeling  and  simulation  at 
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each  activity  visited.  Based  on  cost  data  from  these 
activities  and  extrapolating  from  the  BDM/ITAC  Corporation 
sample  data,  we  estimate  that  during  FY  1992,  DoD 
expenditures  for  modeling  and  simulation  supporting  weapon 
system  development  could  be  as  much  as  $1.3  billion  to 
$1.6  billion  (Appendix  A). 

Our  technical  assistants  estimate  that  a  10-percent 
reduction  in  modeling  effort  can  reasonably  be  achieved 
through  elimination  of  unnecessary  redundancy.  This 
reduction  represents  as  much  as  $803  million  (10  percent  X 
$1,338  billion  X  6  years)  of  unjustified  funds  over  the 
6-year  Future  Years  Defense  Plan. 


Conclusions 

Historically  within  DoD,  models  and  simulations  have  been 
developed  in  response  to  a  specific  question  or  problem. 
Only  rarely  have  these  ad  hoc  models  or  simulations  been 
extended  to  general  solutions  with  an  approach  to  a  wider 
context.  An  important  aspect  of  utilization  is  the 
application  of  a  model  or  simulation  developed  for 
one  particular  use  to  another,  possibly  quite  different, 
use.  The  ability  to  use  a  given  model  across  many 
applications  offers  significant  benefits  by  reducing 
development  effort. 

The  absence  of  interoperability  standards  at  the  OSD  level 
promotes  duplication  and  proliferation  of  computer  models. 
As  these  models  and  simulations  cannot  effectively 
interface,  it  is  difficult  to  use  them  for  analysis  related 
to  joint  operations.  There  is  currently  a  joint  effort 
between  DMSO  and  the  Defense  Advanced  Research  Projects 
Agency  (DARPA)  to  develop  and  implement  the  Aggregate  Level 
System  Protocol  (ALSP) .  ALSP  would  allow  Service-developed 
wargames  to  communicate  and  interoperate.  On  two  major 
exercises  the  ALSP  has  been  successfully  demonstrated. 

There  are  inadequate  internal  controls  relative  to  modeling 
and  simulation  activities  in  the  Military  Departments  and 
Defense  agencies.  Effective  policies,  procedures,  and 
guidelines  to  control  investment  in  modeling  and  simulation 
have  not  been  established. 

Complete  inventories  of  models  and  simulations  and.  their 
respective  capabilities  are  known  only  within  an  individual 
mission  area.  There  is  no  single  authoritative  library  that 
users  and  developers  of  models  can  access  to  determine  if  an 
existing  asset  could  satisfy  a  perceived  need. 

Draft  DoD  Directive  50XX.XX  does  not  establish  a  permanent 
OSD  level  management  and  administrative  structure  to  direct 
and  control  modeling  and  simulation  activities  throughout 
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the  DoD.  Responsibility  for  overseeing  development  of  DoD 
policies,  directives,  procedures,  and  interoperability 
standards  and  protocols  is  assigned  to  the  EXCIMS.  The 
EXCIMS  is  required  to  meet  only  twice  each  calendar  year. 

RECOMMENDATIONS  FOR  CORRECTIVE  ACTION 

We  recommend  that  the  Under  Secretary  of  Defense  for 
Acquisition: 

1.  Assign  responsibility  for  development  of  policies  and 
procedures  related  to  investment,  internal  development, 
interoperability  standards,  modification  of  existing  assets, 
and  maintenance  of  catalogues  regarding  modeling  and 
simulation  activities  within  DoD.  Army  Regulation  5-11 
could  provide  the  baseline  for  development  of  these  DoD 
procedures . 

2.  Develop  a  standard  reporting  procedure  for  starting  new 
modeling  and  simulation  efforts  with  a  cost  threshold 
exceeding  $50,000  for  input  into  a  DoD-wide  catalogue  system 
and  require  all  Components  to  comply  with  this  procedure. 

3.  Require  that  DoD  Components  establish  formal  oversight 
responsibilities  and  associated  internal  controls  for 
modeling  and  simulation  activities,  based  on  Army  Regulation 
5-11,  "Army  Model  and  Simulation  Management  Program." 

4.  Establish  and  maintain  a  common  DoD  modeling  and 
simulation  library  with  ready  access  to  modeling  and 
simulation  users  and  developers. 


MANAGEMENT  COMMENTS  AND  AUDIT  RESPONSES 

The  Director  of  Defense  Research  and  Engineering  responded 
for  the  Under  Secretary  of  Defense  for  Acquisition  and 
indicated  that  the  audit  report  was  a  timely  assessment  of 
modeling  and  simulation  efforts  within  the  Department. 
However,  the  Director  of  Defense  Research  and  Engineering 
feels  that  the  report  did  not  fully  acknowledge  the 
Department's  plans  and  efforts  to  rectify  these 
deficiencies.  The  Director  of  Defense  Research  and 
Engineering  concurred  with  Recommendations  A.I.,  A. 2.,  A. 3., 
and  A. 4.  provided  the  audit  report  include  a  statement  that 
corrective  action  is  in  process. 

Audit  Response.  We  consider  the  comments  from  the 
Director  of  Defense  Research  and  Engineering  to  be 
responsive.  We  agree  that  a  number  of  initiatives  have  been 
undertaken  by  the  Department  to  address  issues  raised  in  our 
audit  report.  We  also  agree  that  when  implemented  the  Draft 
DoD  Directive  50XX.XX,  "Modeling  and  Simulation,"  should 
assist  in  improving  DoD  management  of  modeling  and 
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simulation  activities.  We  have  revised  the  report  to  give 
recognition  to  specific  efforts  and  initiatives  begun  by  the 
EXCIMS,  DMSO ,  and  others  that  seek  to  remedy  problems 
identified  during  the  course  of  our  audit. 

The  Joint  Staff  stated  that  the  subject  audit  provides  an 
excellent  assessment  of  the  status  of  DoD  policy  and 
guidance  in  the  area  of  modeling  and  simulations  standards. 

The  Department  of  the  Army  fundamentally  agreed  with  the 
general  thrust  of  the  audit  report  and  generally  concurred 
with  the  recommendations  as  they  address  the  need  for 
integrated  policies  at  the  DoD  level.  However,  the  Army 
feels  that  the  report  contained  substantive  errors  and 
misuses  in  terminology  that  detracted  from  its  overall 
credibility.  Specifically,  the  Army  felt  the  report  did  not 
recognize  the  ongoing  nature  of  activities  making 
incremental  advances  toward  resolving  essentially  technical 
issues;  the  baseline  sample  used  in  the  audit  was  not 
reliable,  complete,  or  current;  and  the  draft  report  uses 
the  term  "operations  research  assets"  as  a  synonym  for 
"modeling  and  simulation." 

Audit  Response.  The  Army's  response  implies  that 
models  are  for  the  modelers  and  that  DoD  management  should 
trust  that  the  modelers  are  the  appropriate  ones  to  address 
the  issues  raised  in  our  audit.  We  believe  this  viewpoint 
fails  to  consider  the  broad  range  of  applications  for 
modeling  and  simulation  across  the  entire  management 
spectrum  of  the  Department. 

The  Department  of  Defense  and  the  Military  Services  use 
modeling  and  simulation  to  analyze  a  wide  range  of  problems 
and  reasonably  expect  plausible  solutions  to  be  developed 
from  these  analyses.  Decisionmakers  who  use  modeling  and 
simulation  analyses  are  unlikely  to  be  "professional  model 
developers"  and,  accordingly,  they  would  be  unfamiliar  with 
terms,  assumptions,  and  limitations  of  the  "modeling  and 
simulation  community."  Even  though  decisions  that  used 
these  analyses  are  often  significant  and  important,  no 
disclosures  are  required  or  are  typically  made  by  model 
developers  regarding  the  appropriateness  or  accuracy  of 
modeling  and  simulation  in  these  analyses. 

Accordingly,  we  believe  that  decisions  based  on  flawed 
analyses  (due  to  improper  application  of  a  model  or  the  use 
of  a  model  with  an  algorithm  that  is  inherently  flawed)  can 
be  worse  than  decisions  based  on  educated  guesses  and 
intuition.  In  the  former  case,  we  tend  to  ascribe  an  aura 
of  "science"  to  the  decision;  in  the  latter  case,  we 
understand  the  "educated  guess"  element. 
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We  realized  that  the  baseline  sample  used  in  the  audit  was 
not  reliable,  complete,  or  current  and  started  the  audit 
with  the  knowledge  that  no  single,  authoritative  list  of 
models  and  simulations  exists  within  DoD.  This  was  done 
because  one  of  our  main  audit  objectives  was  to  illustrate 
this  lack  of  visibility  at  the  DoD  level  and  to  demonstrate 
the  lack  of  management  control  over  the  DoD  modeling  and 
simulation  effort.  Accordingly,  our  approach  was  to  use  a 
relatively  clean  and  up-to-date  list  to  direct  us  to 
organizations  that  do  use  models  for  weapons  system 
development.  Audit  site  visits  uncovered  significant 
numbers  of  additional  models  and  helped  identify  those 
models  no  longer  in  use.  Since  we  did  not  use  this  sample 
to  project  audit  results  statistically,  the  fact  that  the 
sample  was  incomplete  and  unreliable  is  material  only  to 
further  illustrate  our  point  that  DoD  needs  to  improve 
management  controls  in  modeling  and  simulation. 

References  in  the  draft  audit  report  labelled  "operations 
research  assets"  have  been  changed  to  read  "modeling  and 
simulation"  in  accordance  with  generally  accepted  DoD  usage. 

The  Department  of  the  Naw  did  not  formally  respond. 
However,  informally  the  Navy  agreed  there  is  a  need  for 
centralized  direction  and  control  of  model  and  simulation 
policy.  In  addition,  the  Navy  has  started  to  identify 
duplicative  models;  to  establish  an  inventory  of  current 
models;  and  to  establish  a  master  plan  and  an  investment 
plan. 

The  Naval  War  College  concurred  with  all  recommendations  to 
control  procurement  and  development  of  models  and 
simulations.  To  provide  a  greater  degree  of  assets 
reporting,  however,  the  Naval  War  College  suggested  that  the 
reporting  threshold  criteria  be  lowered  from  the  recommended 
$500,000. 

Audit  Response.  In  our  draft  report  we  had  recommended 
a  cost  threshold  of  $500,000  for  input  into  a  DoD-wide 
catalogue  system.  The  Naval  War  College  suggested  that  the 
reporting  threshold  criteria  be  lowered  from  the  recommended 
$500,000.  After  reviewing  draft  report  comments,  we  agree 
and  have  modified  our  recommendation  to  lower  the  reporting 
threshold  to  $50,000. 

The  Department  of  the  Air  Force  did  not  respond. 
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B.  VERIFICATION.  VALIDATION.  AND  ACCREDITATION 


The  vast  majority  of  models  and  simulations  currently  used 
in  DoD  have  not  been  verified,  validated,  or  accredited. 
This  occurred  because  DoD  does  not  have  a  requirement, 
criterion,  or  standard  by  which  to  accomplish  a 
verification,  validation,  or  accreditation  process.  As  a 
result,  weapon  system  requirements,  procurement  decisions, 
engineering  designs,  test  and  evaluation  results,  and 
operations  may  be  based  on  data  that  is  inaccurate  or 
misleading.  Also  users'  lack  of  confidence  in  models  and 
simulations  that  are  not  verified,  validated,  or  accredited 
encourages  duplication  and  proliferation. 


DISCUSSION  OF  DETAILS 


Background 

Verification  is  the  process  of  determining  whether  a  model 
or  simulation  accurately  represents  the  developer's 
conceptual  description  and  specifications.  Elements  of 
verification  include  determination  of  the  model's  logic  and 
code  and  an  assessment  of  documentation  and  code  versus  a 
standard. 

Validation  is  the  process  of  determining  the  degree  to  which 
a  model  or  simulation  is  an  accurate  representation  of  the 
"real  world"  from  the  perspective  of  the  intended  uses  of 
the  model.  Elements  of  validation  include  face  validation 
(expert  opinion),  input  data  validation  (intelligence),  and 
output  data  validation  (testing  and  sensitivity  analysis) . 

Accreditation  is  an  official  determination  that  a  model  or 
simulation  is  acceptable  for  a  specific  purpose.  Elements 
of  accreditation  include  determining  if  the  model  was 
assessed  under  the  conditions  of  the  analysis  and  how  much 
validation  is  adequate  for  it  to  be  used  in  a  particular 
application. 

DoD  does  not  require  that  models  and  simulations  be 
verified,  validated,  or  accredited.  Although  the  Army  has 
defined  W&A  in  its  Army  Regulation  5-11,  at  the  OSD  level, 
as  well  as  other  Military  Departments  and  Defense  agencies, 
there  are  no  accepted  definitions  for  these  terms;  and  there 
are  no  criteria  or  standards  by  which  W&A  should  be 
accomplished.  The  Navy  and  Air  Force  have  not  developed  a 
requirement,  criterion,  or  standard  by  which  W&A  should  be 
accomplished . 
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Audit  Sample  Results 


At  the  24  locations  visited,  our  technical  experts  evaluated 
62  models  for  W&A  (Appendix  D)  .  Only  three  models  in  this 
sample  (5  percent)  had  completed  a  formal  W&A  process. 
Comparison  studies  or  an  informal  W&A  process  had  been 
accomplished  on  33  models;  the  remaining  26  models 
(42  percent)  had  no  W&A. 

For  comparative  purposes,  we  evaluated  an  additional  sample 
of  26  of  the  more  recently  developed  models  and  found  that 
they  too  had  not  received  formal  W&A.  We  concluded  that 
there  had  been  no  significant  increase  in  the  number  of 
recently  developed  models  that  had  completed  a  formal  W&A. 
We  attribute  this  lack  of  improvement  in  recently  developed 
models  to  the  absence  of  a  DoD  requirement  for  W&A. 

Decisions  Based  on  Model  and  Simulation  Data 

Output  data  from  models  and  simulations  is  frequently  used 
in  the  DoD  decisionmaking  process  to  define  weapon  system 
requirements,  influence  procurement  decisions,  aid  in 
engineering  design,  determine  test  and  evaluation  results, 
and  perform  operations  analyses.  Our  audit  results  indicate 
that  as  many  as  95  percent  of  the  models  used  for  these 
purposes  have  not  been  fully  verified,  validated,  or 
accredited.  Accordingly,  there  is  considerable  risk  to 
users  that  output  data  might  be  inaccurate  and,  therefore, 
lead  to  costly  mistakes  and  delays. 


Verification.  Validation,  and  Accreditation  Not  Required  by 
DoD 


Models  and  simulations  within  the  DoD  have  been  developed  by 
Components  addressing  specific  mission-related  issues  and 
questions.  Models  used  for  these  purposes  typically  had  no 
visibility  at  the  OSD  level. 

Recent  advances  in  computer  technology  allow  computer 
modelers  to  address  issues  and  share  information  to  an 
extent  that  was  formerly  not  possible.  These  advances  have 
expanded  areas  appropriate  for  the  application  of 
modeling/simulation  into  military  operations,  production  and 
logistics,  research  and  development,  and  test  and 
evaluation. 

We  were  unable  to  identify  any  formal  OSD-level  direction, 
guidance,  criteria,  or  standards  regarding  W&A  of  models 
and  simulations,  although  many  DoD  Components  have  realized 
the  need  for  W&A  and  have  initiated  efforts  to  accomplish 
W&A  at  the  Component  level.  These  efforts  have  been 
frustrated  due  to  the  lack  of  a  W&A  requirement  and  the 
absence  of  OSD-level  criteria  and  guidance.  Recent 
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recognition  of  the  pervasive  use  of  modeling  and  simulation 
throughout  the  DoD  for  all  types  of  analyses  have  given 
impetus  to  the  need  for  a  DoD  perspective  regarding 
management  of  this  activity,  in  general,  and  the  need  for 
W&A  of  models  and  simulations,  in  particular. 

Most  major  acquisition  programs  use  modeling  to  verify  one 
or  more  aspects  of  system  performance.  If  a  program  fails 
to  meet  performance  goals,  it  is  difficult  to  identify  any 
single  element  as  the  primary  cause.  The  use  of  models  and 
simulations  that  have  not  been  verified,  validated,  or 
accredited  is  suspected  of  contributing  to  some  performance 
failures.  Specifically,  the  use  of  inaccurate  and  incorrect 
modeling  and  simulation  data  for  decisions  has  contributed 
to  significant  problems  in  DoD.  For  example,  the  B-1B 
Bombers'  defective  Defensive  Avionics  System  was  developed 
with  models  that  were  not  verified,  validated,  and 
accredited.  The  Advanced  Medium  Range  Air-to-Air  Missile 
development  also  utilized  a  model  that  was  not  verified, 
validated,  and  accredited.  Serious  trajectory  problems  had 
to  be  corrected.  Also,  before  the  Transportation  Model 
(TRANSMO)  was  verified,  validated,  and  accredited  in  August 
1991,  it  contained  erroneous  input  data.  The  result  was  a 
significant  overestimate  of  deployment  capability  for 
Operations  Desert  Shield/Storm, 

B-1B  Bomber.  The  AN/ALQ-161  Defensive  Avionics  Suite 
in  the  Air  Force  B-1B  Bomber  was  designed  to  detect, 
identify,  and  classify  hostile  radar  threats  and 
automatically  direct  appropriate  jamming  responses  against 
these  threats  in  a  descending  order  of  priority.  Our  review 
found  that  models  and  simulations  that  were  not  verified, 
validated,  or  accredited  were  used  in  development  and 
testing  of  the  AN/ALQ-161. 

The  prime  contractor  on  the  B-1B  program  developed  a  model 
to  determine  the  probability  of  the  B-lB's  survival .  in  a 
nuclear  war.  Output  data  from  this  model  was  influential  in 
arriving  at  major  program  decisions. 

A  major  subcontractor  on  the  B— IB,  who  developed  the  AN/ALQ- 
161  Defensive  Avionics  Suite,  used  simulations  in  designing 
the  system.  We  were  unable  to  obtain  any  records 
documenting  W&A.  Also,  personnel  familiar  with  the  B— IB 
program  that  we  interviewed  did  not  recall  efforts  being 
made  by  the  subcontractor  for  W&A  of  the  models  and 
simulations.  As  a  result,  the  simulations  ultimately  did 
not  model  high-power  threats  existing  on  test  ranges. 

Specifically,  when  the  AN/ALQ-161  Defensive  Avionics  Suite 
was  flight-tested,  the  "real"  test  operating  environment 
overpowered  the  system.  The  system  could  identify  the 
top  ten  threats  in  a  low-threat  environment  but  would  be 
overwhelmed  in  a  high-threat  environment. 
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On  June  17,  1992,  the  Air  Force  disclosed  that  it  would  cost 
an  estimated  $1.1  billion  to  correct,  enhance,  and  support 
the  B-1B  Defensive  Avionics  Suite. 

AIM-120  AMRAAM  Missile.  The  Advanced  Medium  Range  Air- 
to-Air  Missile  (AMRAAM)  is  a  joint  Air  Force  and  Navy 
program  to  develop  a  medium-range,  radar-guided  missile  to 
be  used  in  air-to-air  combat  against  enemy  aircraft. .  The 
AMRAAM  is  designed  to  be  compatible  with  the  latest  fighter 
aircraft,  including  the  F-14,  F-15,  F-16,  F/A-18,  and  the 
Advanced  Tactical  Fighter. 

The  AMRAAM  missile  test  plan  provides  for  a  combination  of 
simulation  tests,  flight  tests,  and  live  missile  firings. 
Simulations  were  conducted  before  live  missile  firings  to 
predict  actual  missile  performance  and  examine  other 
scenarios  otherwise  too  dangerous  or  costly  to  perform  or 
both.  The  AMRAAM- Installed  System  Performance  Reliability 
(ISPR)  is  a  weapon  system  performance  model  used  to  support 
Operational  Test  and  Evaluation  (OT&E)  of  the  AMRAAM 
missile. 

One  function  of  AMRAAM-ISPR  is  to  simulate  an  AMRAAM  missile 
six-degree-of-motion  trajectory  after  launch.  Before  1988, 
earlier  versions  of  this  model  that  had  not  been  verified, 
validated,  or  accredited  were  used  for  OT&E  purposes. 
Subsequently,  when  live  missile  firing  test  results  were 
analyzed,  the  results  indicated  serious  problems  with  the 
six-degree-of-motion  trajectory  after  launch. 

The  six-degree-of-motion  trajectory  after  launch  problem  had 
not  been  detected  by  simulations  run  before  live  firing 
tests.  We  believe  that  a  model  subjected  to  a  rigorous  W&A 
process  would  have  provided  data  with  a  greater  degree  of 
reliability  and  accuracy  and  might  have  detected  this 
trajectory  problem  before  conducting  live  firing  tests. 

In  1988,  a  later  version  of  the  AMRAAM-ISPR  model  was 
validated  but  not  verified  or  accredited.  Nevertheless, 
data  provided  by  these  models  has  been  used  by  the  Defense 
Acquisition  Board  to  support  decisions  made  regarding  the 
AMRAAM  program.  In  addition,  because  the  trajectory  problem 
was  not  detected  until  after  live  firing  tests  were 
conducted,  AMRAAM  missile  software  had  to  be  modified  to 
correct  the  problem. 

TRANSMO .  The  Transportation  Model  is  a  time-oriented, 
event-stepped  simulation  of  inter-theater  deployment 
activities  based  on  availability  of  transportation  assets 
and  prioritized  requirements  for  deployment.  The  model 
computes  arrival  schedules  for  units  and  resupply  cargo. 
The  TRANSMO  was  not  formally  accredited  and  was  not  verified 
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and  validated  until  August  1991  when  data  collected  during 
Operations  Desert  Shield/Storm  (ODS)  was  used  to  compare 
TRANSMO-predicted  results  to  ODS  actual  results. 

Specifically,  for  cargo  delivered  by  sea,  the  simulated 
deployment  was  in  error  by  84  percent;  for  cargo  delivered 
by  air,  the  simulated  deployment  error  increased  to 
178  percent.  Overall,  the  TRANSMO  simulation  underestimated 
the  total  cargo  time  to  a  Middle-East  theater  of  operations 
by  114  percent. 

Deployment  capability  estimates  by  TRANSMO  were 
significantly  overestimated  because  of  optimistic  planning 
data  which  contained  inaccurate  scenario  assumptions  and 
planning  factors.  If  the  TRANSMO  simulation  had  been 
subjected  to  a  rigorous  W&A  process  before  ODS,  inaccurate 
input  data  could  have  been  reviewed  and  corrections  could 
have  been  made,  resulting  in  a  greater  degree  of 
reliability. 

Compounding  of  Errors 

It  is  important  to  note  that  data  outputs  from  one  model  or 
simulation  are  frequently  used  as  data  inputs  to  a  second  or 
perhaps  even  a  third  model  or  simulation  in  a  chain-linked 
series.  Accordingly,  if  erroneous  output  data  is  used  from 
a  model  or  simulation  lacking  W&A,  errors  are  likely  in  all 
subsequent  analyses. 

Confidence  Placed  in  Modeling/Simulation  Data 

Even  though  a  given  model/ simulation  has  not  been  subjected 
to  a  rigorous  W&A  process,  analyses  performed  by  these 
models  frequently  assume  a  significant  role  in  DoD  decision¬ 
making.  A  specific  case  is  a  model  named  "Survivability  by 
Computer  Analysis"  (SCAN) .  It  models  performance  of  the 
missile  warhead's  proximity  fuze,  fragmentation  of  the 
missile  warhead,  and  target  damage  caused  by  individual 
warhead  fragments.  SCAN  produces  a  probability  of  target 
kill  by  tracing  the  paths  of  warhead  fragments  through  the 
target. 

Although  we  were  told  that  lethality  analysts  have 
informally  verified  SCAN  code  against  actual  fuze  software 
and  design  documents,  we  could  not  obtain  documentation  to 
support  any  comparisons  between  results  obtained  with  the 
SCAN  model  and  actual  missile  flight  tests.  Despite  this 
lack  of  W&A,  SCAN  has  been  used  to  support  operational 
evaluations  of  the  AIM-120  (AMRAAM) ,  the  AIM-7  (SPARROW), 
the  AIM-9  (SIDEWINDER) ,  the  AIM-54  (PHOENIX) ,  and  the 
PATRIOT  missiles. 
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In  1991,  a  major  Defense  contractor  requested  a  copy  of  SCAN 
to  use  on  modeling  of  air-to-ground  hardened  target 
penetrators  for  design  and  concept  proposals  and  the  Army 
requested  a  copy  of  SCAN  to  use  in  vulnerability  modeling  of 
two  amphibious  alternatives  for  the  Bradley  fighting 
vehicle.  In  addition,  SCAN  has  also  been  used  for  battle 
damage  and  repair  studies  for  F-16  and  C-130  aircraft. 

Users  have  placed  confidence  in  the  SCAN  analyses  related  to 
research,  development,  and  test  and  evaluation  of  major 
weapon  system  programs  even  though  the  veracity  of  these 
analyses  has  not  been  demonstrated.  As  a  result,  the 
Defense  Acquisition  Board  and  other  acquisition  officials 
may  have  based  significant  acquisition  decisions  on  analyses 
generated  by  a  model  that  has  not  been  verified,  validated, 
or  accredited. 

Conclusions 


Data  generated  by  models  and  simulations  are  used  in 
analyses  to  support  decisions  of  significant  value  and 
importance.  However,  model  developers  and  users  have  not 
been  performing  W&A  because  there  was  no  DoD  requirement  or 
standard  by  which  W&A  was  to  be  done.  In  addition,  the 
absence  of  DoD  criteria  resulted  in  W&A  having  low  priority 
and  inadequate  justification  for  expenditure  of  funds  and 
allocation  of  resources. 

Audit  results  indicate  95  percent  of  models  and  simulations 
used  in  DoD  have  not  been  subjected  to  a  formal  W&A 
process.  Using  these  models  can  result  in  erroneous  and 
misleading  data  being  introduced  to  the  decisionmaking 
process.  In  the  case  of  major  weapon  system  development 
programs,  this  may  lead  to  large  additional  expenditures  to 
correct  errors  introduced  by  using  inaccurate  data. 

A  DoD  requirement  for  W&A  of  models  and  simulations  can 
eliminate  or  reduce  errors  in  computer  modeling  data  that  is 
used  by  DoD  personnel  in  the  decisionmaking  process.  The 
W&A  process  adds  assurance  and  confidence  that  accurate 
input  data  is  used  in  models  that  will  contribute  to  the 
prevention  of  costly  and  time-consuming  mistakes. 

Draft  DoD  Directive  50XX.XX  assigns  DoD  Components  the 
responsibility  for  development  of  W&A  policies  and 
procedures  for  modeling  and  simulation  applications  managed 
by  the  Component.  It  also  assigns  to  the  Component  the 
responsibility  for  the  accreditation  of  modeling  and 
simulation  applications  used  to  support  major  DoD 
decisionmaking,  such  as  the  Defense  Acquisition  Board,  Joint 
Requirements  Oversight  Council,  and  the  Defense  Planning  and 
Resources  Board.  Nowhere  in  this  directive  is  the  role  of 
the  Defense  Intelligence  Agency  addressed  with  respect  to 
the  W&A  of  threat  forces  and  capabilities  in  models  and 
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simulations.  In  addition,  we  feel  that  the  policy, 
guidance,  standards,  and  criteria  by  which  W&A  is  to  be 
accomplished  <should  be  established  at  the  OSD  level  for  the 
guidance  and  direction  of  DoD  Components. 


RECOMMENDATIONS  FOR  CORRECTIVE  ACTION 

We  recommend  that  the  Under  Secretary  of  Defense  for 
Acquisition: 

1.  Using  the  Army  Model  and  Simulation  Management  Program 
as  a  framework,  develop  policy,  guidance,  standards,  and 
criteria  by  which  verification,  validation,  and 
accreditation  of  models  and  simulations  is  to  be 
accomplished. 

2.  Require  that  models  and  simulations  used  for 
requirements  definition,  weapon  system  development, 
engineering,  and  testing  and  evaluation  be  verified, 
validated,  and  accredited  in  accordance  with  policy, 
guidance,  standards,  and  criteria  of  Recommendation  1. 


MANAGEMENT  COMMENTS  AND  AUDIT  RESPONSES 

The  Director  of  Defense  Research  and  Engineering,  responding 
for  the  Under  Secretary  of  Defense  for  Acquisition, 
concurred  with  Recommendations  B.l.  and  B.2.,  provided  the 
audit  report  include  a  statement  that  corrective  action  is 
in  process.  The  Director  of  Defense  Research  and 
Engineering  (DDR&E)  noted  that  the  Draft  DoD  Directive 
50XX.XX,  "Modeling  and  Simulation,"  establishes  requirements 
for  verification,  validation,  and  accreditation  of  models 
and  simulations.  The  Director  also  noted,  however,  that  it 
would  be  very  costly  to  prescribe  an  enforceable  single  W&A 
standard  and  that  DDR&E  feels  that  the  W&A  process  needs  to 
be  defined  by  the  individual  DoD  component. 

Audit  Response.  We  consider  the  comments  from  the 
Director  of  Defense  Research  and  Engineering  to  be  generally 
responsive.  However,  we  still  believe  that  a  single  W&A 
standard  needs  to  be  defined  at  the  OSD  level.  We  feel  that 
there  are  many  instances  where  general  use  modeling  and 
simulation  applications  may  be  used  to  support  major  DoD 
decisionmaking  organizations  and  processes  such  as  the 
Defense  Planning  and  Resources  Board;  the  Defense 
Acquisition  Board;  the  Joint  Requirements  Oversight  Council; 
and  the  DoD  Planning,  Programming,  and  Budgeting  system. 

Draft  DoD  Directive  50XX.XX  assigns  management 
responsibility  for  these  general  use  models  and  simulations 
to  a  DoD  Component  designated  as  the  "DoD  Modeling  and 
Simulation  Executive  Agent"  for  that  general  use 
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application.  The  policies  and  procedures  used  to  W&A  that 
general  use  model  or  simulation  would  be  the  ones 
established  by  that  respective  DoD  Component.  Therefore, 
unless  a  single  minimum  standard  for  W&A  is  established  at 
the  OSD  level,  each  respective  general  use  model  or 
simulation  used  to  support  these  major  DoD  decisionmaking 
organizations  may  have  been  verified,  validated,  and 
accredited  by  different  standards.  Accordingly,  we  request 
that  the  Director  of  Defense  Research  and  Engineering 
reconsider  his  position  when  responding  to  the  final  report. 

The  Joint  Staff  stated  that  the  subject  audit  provides  an 
excellent  assessment  of  the  status  of  DoD  policy  and 
guidance  for  modeling  and  simulation  verification, 
validation,  and  accreditation. 

The  Department  of  the  Army  indicated  that  the  definitions  of 
verification,  validation,  and  accreditation  and  the  methods 
used  to  apply  them  to  models  and  simulations  are  much 
debated  topics  within  the  community.  The  lack  of  a  specific 
DoD  mandate  was  not  the  reason  that  developers  do  not  seem 
to  do  an  adequate  job  in  this  area.  The  Army  felt  quite  the 
contrary,  that  there  is  much  serious  discussion  and  debate 
about  what  constitutes  adequate  verification,  validation, 
and  accreditation  and  how  and  when  to  apply  the  various 
techniques  that  are  available. 

Audit  Response.  The  response  by  the  Army  indicates  a 
situation  where  those  responding  may  have  too  narrow  a  view 
of  where,  when,  and  how  models  are  being  applied  within  DoD. 
Our  point  is  that  most  people  who  use  models  in  the 
Department  see  only  how  they  themselves  use  them;  they  fail 
to  understand  the  broad  range  of  applications  for  models 
across  the  entire  management  spectrum  of  the  Department. 

We  realize  that  there  is  much  serious  discussion  and  debate 
about  what  constitutes  adequate  verification,  validation, 
and  accreditation  and  how  and  when  to  apply  the  various 
techniques  that  are  available.  However,  after  more  than 
40  years,  there  is  still  no  agreement  in  the  "professional 
modeling  and  simulation  community"  as  to  a  generally 
accepted  definition  of  verification,  validation,  and 
accreditation  or  how  it  should  be  accomplished. 

Currently,  the  weaknesses  and  limitations  of  specif ic  models 
and  simulations  are  known  just  by  a  handful  of  "modeling  and 
simulation  professionals."  The  proposed  verification, 
validation,  and  accreditation  requirement  will  help  ensure 
that  the  strengths,  weaknesses,  and  limitations  of  models 
and  simulations  will  be  disclosed  up  front,  to  the 
decisionmakers . 


The  Department  of  the  Navy  did  not  formally  respond. 
However,  informally  the  Navy  agreed  there  is  a  need  for 
centralized  policy  direction  related  to  verification, 
validation,  and  accreditation.  In  addition,  the  Navy  has 
started  to  develop  instructions  and  procedures  related  to 
the  verification,  validation,  and  accreditation  of  models 
and  simulations. 

The  Naval  War  College  concurred  with  all  recommendations  to 
establish  plans,  policies,  and  procedures  relative  to 
verification,  validation,  and  accreditation  of  models  and 
simulations. 

The  Department  of  the  Air  Force  did  not  respond. 
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C.  CONFIGURATION  MANAGEMENT  AND  DOCUMENTATION 


The  majority  of  models  and  simulations  used  in  DoD  lack 
adequate  configuration  management  and  documentation 
necessary  to  assure  readily  available  access  by  authorized 
Defense  personnel.  This  condition  exists  because  of 
ineffective  guidance  and  lack  of  oversight  by  Department  of 
Defense  Components.  As  a  result,  users  and  developers  of 
models  and  simulations  are  unwilling  or  unable  to  use 
existing  computer  models,  which  can  result  in  added  costs  to 
modify  present  models  and  simulations  or  develop  new  ones. 


DISCUSSION  OF  DETAILS 

Background 

Configuration  management,  as  defined  by  DMSO,  is  '‘The 
application  of  technical  and  administrative  direction  and 
surveillance  to  identify  and  document  the  functional  and 
physical  characteristics  of  a  model,  control  changes,  and 
record  and  report  change  processing  and  implementation 
status . " 

Documentation  refers  to  items  such  as  programmer's  manuals, 
user's  manuals,  analyst's  manuals,  and  other  media  that 
assist  or  instruct  the  user  in  the  operation  or  maintenance 
of  a  model /simulation. 

Inadequate  configuration  Management  and  Documentation 

We  found  that  agency -wide  configuration  management  and 
documentation  of  models  and  simulations  were  usually 
nonexistent  or  requirements  were  not  enforced.  Several 
activities  had  inconsistent  results  even  at  the  same 
location. 

Specifically,  some  models  and  simulations  reviewed  had 
adequate  configuration  management  and  were  we 11 -documented 
while  others  had  no  configuration  management  or 
documentation.  At  one  Air  Force  activity,  we  reviewed 
six  models  and  found  that  two  models  had  good  configuration 
management  programs  and  another  two  models  were  well- 
documented.  However,  of  the  six  models/simulations 
reviewed,  none  had  both  adequate  configuration  management 
and  good  documentation. 

Similarly,  review  of  two  models  in  use  at  a  Navy  test 
facility  found  that  one  had  adequate  configuration 
management  and  was  completely  documented,  while  the  other 
model  reviewed  had  no  elements  of  an  effective  configuration 
management  program  and  had  no  documentation.  Similar 
situations  were  also  found  at  Army  activities  (Appendix  E) . 
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Discussions  with  model  developers  and  users  at  these 
locations  disclosed  that  the  lack  of  a  clear  requirement  or 
guidance  from  DoD  or  the  Component  headquarters  caused  this 
situation.  Accordingly,  configuration  management  policy  and 
the  degree  of  required  documentation  for  models  and 
simulations  is  being  determined  by  each  model  developer  on  a 
model-by-model  basis. 

Configuration  Management 

Changes  to  coding  of  a  model  could  have  adverse  effects  on 
model  performance.  Configuration  management  of  models  and 
simulations  is  important  because  only  by  managing  changes  to 
models/simulations  that  have  been  verified,  validated,  and 
accredited  will  potential  users  be  assured  that  a  model  or 
simulation  will  perform  as  intended.  Effective  configuration 
management  includes  the  planning,  documenting,  and  reporting 
of  any  changes  or  modifications  and  the  effect  of  the  change 
or  modification  on  model  performance  and  use. 

Criteria.  Three  elements  are  necessary  for  adequate 
configuration  management.  Specifically,  these  elements  are: 

o  Configuration  management  policy  which  estab¬ 
lishes  the  administrative  process  for  approving  and 
documenting  changes  to  the  model  or  simulation; 

o  Configuration  management  plans  that  describe 
how  changes  to  the  existing  model  or  simulation  will  be 
accomplished;  and 

o  Configuration  management  board  or  official  with 
prior  approval  authority  for  all  proposed  changes  to  the 
model  or  simulation. 

Audit  sample  Results.  We  reviewed  62  models  and 
simulations  during  our  audit  for  adequacy  of  configuration 
management.  We  found  that  only  16  (26  percent)  of  these  had 
adequate  configuration  management  control  procedures  in 
place.  While  35  models  or  simulations  had  some  formal  or 
informal  configuration  management,  27  (44  percent)  of  those 
reviewed  had  none  of  the  essential  elements  in  place  for 
effective  configuration  management  (Appendix  E) . 

At  several  activities,  we  noted  that  models  and  simulations 
developed  ,,in-house"  generally  had  less  formal  configuration 
management  control  procedures  than  did  models  and 
simulations  developed  by  Government  contractors.  In 
addition,  more  recently  developed  models  and  simulations 
were  more  likely  to  have  adequate  configuration  management 
procedures.  Still,  only  35  percent  of  recently  developed 
models  and  simulations  had  adequate  configuration  management 
versus  22  percent  of  the  earlier  ones. 
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Documentat ion 


Documentation  explains  how  and  for  what  purposes  a  model  or 
simulation  can  be  used.  Inadequate  or  inaccurate 
documentation  can,  therefore,  result  in  a  model  or 
simulation  being  useless  to  aftyone  other  than  the  original 
developer . 

Criteria.  Military  Standard  2167A  establishes 
documentation  requirements  for  software  developed  for  or  by 
the  DoD.  We  believe  that,  as  a  minimum,  documentation  for 
models  and  simulations  should  consist  of  a  user's  manual, 
analyst's  manuals,  and  programmer's  manuals. 

Audit  sample  Results.  We  reviewed  available 
documentation  for  the  62  models  or  simulations  in  our  sample 
and  found  that  only  8  (13  percent)  had  adequate 
documentation  based  on  this  criteria.  Of  the  remaining 
54  models,  20  had  two  of  the  three  required  documents.  One 
of  the  three  required  documents  was  available  for  14  models. 
However,  20  models  (32  percent)  had  no  documentation 
(Appendix  E) . 

While  our  analysis  indicates  that  more  recently  developed 
models  and  simulations  have  a  higher  percentage  of  minimally 
acceptable  documentation  (14  percent  versus  12  percent), 
this  difference  is  considered  statistically  insignificant. 

As  with  configuration  management,  we  noted  that  computer 
models  developed  "in-house"  are  much  less  likely  to  be  well- 
documented  than  those  developed  for  the  Government  by 
contractors.  The  reason  most  often  cited  for  this  disparity 
is  that  in-house  developers  do  not  expect  their  models  to  be 
used  by  others.  In-house  developers  do  not  usually  conform 
to  the  DoD  Standard  2 167 A  requirements  for  documentation. 


Current  Initiatives 

There  have  been  several  recent  efforts  undertaken  to  address 
modeling  and  simulation  standards  related  to  documentation 
and  configuration  management.  Specifically,  these  include 
the  Aggregate  Level  Simulation  Protocol  (ALSP),  the 
Distributed  Interactive  Simulation  (DIS) ,  and  the  Defense 
Simulation  Internet  (DSI) .  ALSP  was  started  in  1990  by  the 
Defense  Advanced  Research  Projects  Agency  (DARPA)  to  design 
an  interface  for  multiple  combat  simulations.  It  is 
currently  chartered  by  USD (A)  through  the  EXCIMS.  ALSP  is 
developing  field  interface  protocols  and  supports  an 
integrated  multi-function  training  environment  for  joint  and 
combined  exercises  through  configuration  management  of  ALSP 
protocols  and  system  software,  and  deployment  of  tools  and 
documentation . 
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DIS  is  an  effort  to  develop  a  common  standard  for  describing 
the  form  and  types  of  messages  to  be  exchanged  between 
simulated  entities.  DIS  is  another  DARPA  effort  sponsored 
by  the  DMSO.  The  Institute  of  Electrical  and  Electronic 
Engineers  (IEEE)  have  issued  this  common  standard  as  IEEE 
Draft  Standard  P1278. 

DSI  is  intended  to  be  a  high  performance,  wide  area,  packet 
switched  network  to  support  the  infrastructure  for  a  DoD 
seamless  warfighting  simulation.  The  Defense  Information 
Systems  Agency  (DISA)  and  DARPA,  with  support  from  the  DMSO 
are  developing  DIS.  The  configuration  management  controls 
and  standardization  embodied  by  the  ALSP,  DIS,  and  DSI 
initiatives  will  enable  DoD  Components  to  utilize  modeling 
and  simulation  in  support  of  training,  acquisition,  and 
mission  support  at  significantly  less  cost  than  that 
required  for  separate  simulations. 


Conclusion 


Analysis  of  sample  data  and  discussions  with  developers  and 
users  of  models  and  simulations  indicate  that  DoD  management 
controls  regarding  configuration  management  and 
documentation  are  inadequate.  Our  audit  sample  results 
indicated  that  only  26  percent  of  the  models/simulations 
reviewed  had  adequate  configuration  management  control 
procedures  in  place.  Although  most  models/simulations  in 
the  sample  had  some  form  of  documentation,  few  had 
sufficient  documentation  to  make  the  model  reasonably 
accessible  to  potential  users. 

Documentation  and  configuration  management  .  control 
procedures  that  are  inadequate  lead  to  costly  duplication  of 
effort.  Potential  users  may  feel  that  developing  a  new 
model  or  simulation  is  less  risky  than  attempting  to  use  an 
existing  one  that  is  inadequately  documented. 


RECOMMENDATIONS  FOR  CORRECTIVE  ACTION 

We  recommend  that  the  Under  Secretary  of  Defense  for 
Acquisition: 

1.  Establish  a  DoD-wide  policy  requiring  configuration 
management  plans  for  all  future  development  of  models  and 
simulations. 

2.  Establish  a  DoD-wide  policy  requiring  adequate 
documentation  for  all  future  development  of  models  and 
simulations. 
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MANAGEMENT  COMMENTS  AND  AUDIT  RESPONSES 

The  Director  of  Defense  Research  and  Engineering,  responding 
for  the  Under  Secretary  of  Defense  for  Acquisition, 
concurred  with  Recommendations  C.l.  and  C.2.,  provided  the 
audit  report  include  a  statement  that  corrective  action  is 
in  process.  The  Director  of  Defense  Research  and 
Engineering  states  that  implementation  of  Draft  DoD 
Directive  50XX.XX,  "Modeling  and  Simulation,"  includes  the 
designation  of  a  configuration  management  proponent  in  each 
modeling  and  simulation  application. 

The  Director  of  Defense  Research  and  Engineering  also  notes 
that  Draft  DoD  Directive  50XX.XX,  "Modeling  and  Simulation," 
has  specific  documentation  requirements  including 
development  of  a  master  plan  and  an  investment  plan. 

Audit  Response.  We  consider  the  comments  from  the 
Director  of  Defense  Research  and  Engineering  to  be 
responsive.  We  agree  that  a  number  of  initiatives  have  been 
undertaken  by  the  Department  to  address  issues  raised  in  our 
audit  report.  We  also  agree  that  when  implemented  the  Draft 
DoD  Directive  50XX.XX,  "Modeling  and  Simulation,"  should 
assist  in  improving  configuration  management  and 
documentation  of  DoD  modeling  and  simulation  activities. 

The  Joint  Staff  did  not  comment  on  this  finding  and 
recommendation . 

The  Department  of  the  Army  stated  that  lack  of  configuration 
management  of  models  and  simulations  and  inadequate 
documentation  were  difficult  problems  that  "have  been  with 
us  a  long  time." 

Audit  Response.  While  Army  concurs  with  the  condition 
and  effect  of  the  finding,  they  see  the  cause  not  as  a  lack 
of  criteria  or  DoD  Directives  but  as  a  long-term  problem  on 
which  incremental  advances  are  being  made.  The  Army  further 
states  that  "bureaucratic  fiat  will  not  resolve  the  issues 
involved."  This  appears  to  contradict  Army  Regulation  5-11, 
"Army  Model  and  Simulation  Management  Program," 
July  10,  1992.  This  Army  Regulation  requires  that 
configuration  management  be  applied  to  any  model  or 
simulation  developed  wholly  or  partly  with  Army  funding. 
We  see  a  DoD  requirement  as  a  means  to  end  incremental 
advances  and  stop  a  40— plus— year  debate  in  the  "modeling  and 
simulation  community . " 

The  Department  of  the  Naw  did  not  comment  on  this  finding 
and  recommendation. 
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The  Naval  War  College  concurred  with  all  recommendations  to 
establish  policies,  plans,  and  procedures  to  effectively 
institute  configuration  management  and  to  obtain 
documentation . 

The  Department  of  the  Air  Force  did  not  respond. 
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PART  III  ~  ADDITIONAL  INFORMATION 


Appendix  A 

Appendix  B 
Appendix  C 

Appendix  D 

Appendix  E 

Appendix  F 

Appendix  G 

Appendix  H 
Appendix  I 


Estimated  DoD  Modeling  and  Simulation 
Expenditures  for  FY  1992 

Model /Simulation  Technical  Questions 

Model  Name  Versus  Functional  Area  Sample 
Results 

Verification,  Validation,  and  Accreditation 
Sample  Results 

Documentation  and  Configuration  Management 
Sample  Results 

Prior  Audits  and  Studies  Related  to  Modeling 
and  Simulation 

Summary  of  Potential  Benefits  Resulting  From 
Audit 

Activities  Visited  or  Contacted 
Report  Distribution 


The  following  acronyms  are  used  in  this  report. 

ajjC  .  Army  Materiel  Command 

ARDEC  ...  Armament  Research,  Development,  and  Engineering  Center, 
Army 

DASCP  ..  Directorate  for  Advanced  Systems,  Concepts,  and  Planning 

DCAA  .  Defense  Contract  Audit  Agency 

DFARS " . Defense  Federal  Acquisition  Regulation  Supplement 

DOE  .  Department  of  Energy 

FAR  1 1 !  1 ! .  Federal  Acquisition  Regulation 

FFRDC  .  Federally  Funded  Research  and  Development  Center 

GAo  .  General  Accounting  Office 

jPL  [ .  Jet  Propulsion  Laboratory 

MICOM  . . . . .  Missile  Command,  Army 

MPRI  .  Military  Professional  Resources,  Incorporated 

NASA  .  National  Aeronautics  and  Space  Administration 

NR0  .  NASA  Resident  Office 

Oig  .  Office  of  the  Inspector  General 

TACOM  .  Tank-Automotive  Command,  Army 

TVA  .  Tennessee  Valley  Authority 


APPENDIX  As  ESTIMATED  DoD  MODELING  AND  SIMULATION 
EXPENDITURES  FOR  FY  1992 


Different  funding  sources  make  it  difficult  to  precisely 
identify  total  DoD  expenditures  related  to  modeling  and 
simulation  efforts  supporting  weapon  system  development. 
Therefore,  our  objective  was  to  obtain  order  of  magnitude 
computations  related  to  the  cost  of  1,549  models  identified 
as  being  owned  and  controlled  by  DoD  and  its  Components. 

The  following  estimates  of  expenditures  are  based  on 
extrapolation  of  data  we  collected  during  the  audit.  These 
estimates  are  not  based  on  the  statistical  sample  and, 
accordingly,  are  not  statistical  estimates.  Rather  they 
represent  the  best  indicators  presently  available  of  DoD 
expenditures  related  to  modeling  and  simulation  for  FY  1992. 

Our  analyses  used  two  approaches  based  on  expenditures 
required  to  operate  the  24  activities  visited  during  the 
audit.  The  Average  Cost  per  Model/Simulation  approach  used 
an  average  cost  per  model  or  simulation,  weighted  by 
activity  and  service.  The  Proportioned  Cost  of  Activities 
Using  Modeling/Simulation  approach  used  a  ratio  of 
activities  on  which  we  had  cost  data  compared  to  total 
activities  identified  in  the  compendium  assembled  by  the 
audit  team. 

Specifically,  for  FY  1992  the  Average  Cost  per 
Model/ Simulation  approach  estimates  total  DoD  expenditures 
at  $1.58  billion.  The  Proportioned  Cost  of  Activities  Using 
Modeling/Simulation  approach  estimates  total  DoD  expend¬ 
itures  at  $1,338  billion. 
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APPENDIX  B!  MODEL /SIMULATION  TECHNICAL  QUESTIONS 


GENERAL  QUESTIONS  REGARDING  EACH  MODEL /SIMULATION 

1.  What  is  the  name  of  the  model/ simulation? 

2.  What  kind  of  model  is  it,  e.g.,  Air  Defense,  1-on-l, 
M-on-N? 

3.  Is  this  a  single  model  or  a  collection  of  models? 

4.  Who  owns  (controls  access  to)  the  model? 

5.  Who  is  the  point-of -contact  for  information  regarding 
the  model? 

6.  When  was  the  model  first  operational? 

7.  Is  the  model  currently  in  use  on  a  regular  basis? 

8.  Is  the  model  a  stand-alone  model  or  does  it  work  with 
other  models? 

8.1.  If  it  can  work  with  other  models,  to  what 
degree  (e.g.,  are  they  interactive  or  do  you 
just  get  output  data  and  then  use  that  data 
as  input  to  a  separate  run  of  another 
model)? 

8.2.  What  are  the  models  that  work  with  your 
model? 

8.3.  When  was  it  done  and  the  name  of  a  point-of- 
contact  to  be  used  to  obtain  information 
regarding  that  model? 

8.4.  If  it  works  with  other  models,  is  there  an 
interface  standards  document? 

8.5.  What  does  the  model  or  simulation  require 
for  input  and  what  is  the  source  of  this 
input? 

9.  How  many  times  is  it  necessary  to  run  the  model  to  get 
reliable  results? 

9.1.  How  was  that  number  derived? 

9.2.  Is  there  a  standard  that  was  used  to  derive 
the  necessary  number  of  runs? 
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(Continued) 


10.  In  developing  (or  updating)  the  model,  what  efforts 
were  made  to  obtain  tri-Service  requirements? 

10.1.  What  efforts  were  made  to  obtain  intra- 
Service  requirements? 

10.2.  What  efforts  were  made  to  obtain 
requirements  from  anyone  outside  of  your 
immediate  organization? 

11.  In  developing  the  model,  what  efforts  were  made  to  use 
existing  algorithms? 

11.1.  Did  the  algorithms  exist? 

11.2.  Where  did  you  look  for  these  algorithms? 

11.3.  In  what  form  were  the  algorithms 
(documentation) ;  did  you  use  them? 

12.  What  is  the  run  time  of  the  model? 

13.  What  does  the  model  simulate  (for  what  purpose  is  it 
used) ? 

13.1.  What  is  the  end  use  for  the  model's  output? 

13.2.  Has  this  model  been  modified  as  the  threat 
has  changed? 

13.3.  Has  threat  data  been  validated  by  the 
Defense  Intelligence  Agency? 

13.4.  Is  there  a  weapons  system  in  the  development 
or  acquisition  cycle  that  this 
model/simulation  is  being  used  for?  If  so, 
identify  this  weapons  system? 
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orTESTTDNS  REGARDING  DEVELOPMENT  OF  EACH  MODEL /SIMULATION 

14. 

When  was 

the  model  developed? 

14.1. 

Who  developed  it? 

14.2. 

How  much  did  it  cost? 

14.3. 

Why  was  it  developed? 

14.4. 

Can  it  support  any  major  acquisition 
decision? 

14.5. 

Was  it  ever  used  to  support  a  major 
acquisition  decision?  Is  there 
documentation  to  support  a  "yes"  answer? 

14.6. 

Did  it  ever  support  any  decisions?  Are  they 
documented? 

15. 

How  long 

did  it  take  to  develop  the  model? 

16. 

Is  any  part  of  the  model  proprietary  (model,  parts  of 

the  model 

,  source  code,  documentation)? 

16.1. 

How  much  will  it  cost  to  buy  the  rights 
(including  documentation  and  source  code)? 

17.  What  programming  standards  were  used  in  the  development 
of  the  model  and/or  the  updates? 


18.  In  what  language (s)  is  the  model  written? 

18.1.  Is  there  a  version-specific  language  used? 

18.2.  Why  was  the  language  used  and  was  this 
decision  documented? 

18.3.  Was  Ada  considered;  was  a  waiver  sought  and 
the  decision  documented? 

18.4.  How  big  is  the  program  (lines  of  code  and 
disk  space)? 
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QUESTIONS  REGARDING  SPECIFIC  CHARACTERISTICS  OF  EACH 

MODEL /SIMULATION 

19.  How  is  the  model  distributed? 

19.1.  Type  media? 

19.2.  Documentation? 

19.3.  Update  notification? 

20.  On  what  computer  is  the  model  presently  running? 

20.1.  What  is  the  domain  of  platforms  (what  are 
the  platforms  the  model  can  run  on;  is  the 
model  touted  as  "machine  independent"  or 
close  to  it  for  a  class  of  computer 
platforms) ? 

21.  What  operating  system  is  required  to  support  the  model? 

21.1.  Is  a  specific  version  required? 

21.2.  What  libraries  are  required  for  the  model  to 
run  (e.g.,  a  specific  graphics  or  utility 
library  of  programs)? 

21.3.  Are  special  commercial  licenses  required? 

22.  If  a  new  user  desires  to  use  the  model,  what  hardware 
and  software  are  required? 

22.1.  Is  anything  else  required? 

22.2.  How  does  the  user  know  that  your  model 
exists  and  what  its  capabilities  and 
limitations  are? 

22.3.  How  does  the  user  get  the  model  and  what 
documentation  comes  with  it? 

22.4.  Is  formal  training  available? 

22.5.  What  is  the  total  cost  to  the  user  for  the 
model? 
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23.  Does  the  model  run  in  real  time  (can  it)? 

23.1.  If  the  model  is  not  a  real  time  model,  what 
are  its  performance  characteristics  (usually 
in  X  times  real  time,  e.g.,  the  model  may 
run  10  to  1  or  the  model  takes  10  seconds  to 
represent  1  second  of  real  time;  high 
fidelity  models  can  be  5000  to  1)? 

24.  What  fidelity  is  the  model  (how  many  levels  are  there)? 


24.1. 

How  do  you  define  that  fidelity  level? 

24.2. 

What  distinguishes  this  fidelity  level  from 
any  other? 

24.3. 

Can  the  model  run  at  more  than  fidelity 
level? 

24.4. 

Can  the  model  interact  with  another  model  at 
the  same  fidelity  level? 

24.5. 

Can  the  model  interact  with  another  model  at 
a  different  fidelity  level? 

Can  the  user  of  the  model  choose  the  scenario  for  the 
simulation? 

• 

H 

• 

in 

CM 

How  much  can  the  user  deviate  from  the 
"classic"  scenario? 

26.  Does  the  model  include  some  type  of  warning  that  it  is 
not  good  for  certain  scenarios? 

26.1.  Are  there  restrictions  on  its  use  for  those 
scenarios? 

27.  Does  the  model  specifically  state  how  it  should  be 
used? 

27.1.  Does  it  state  how  it  should  not  be  used? 

28.  Does  the  model  have  a  post  processing  capability? 

28.1.  What  is  produced  by  the  model  (data, 
graphics,  both)? 
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29.  Can  you  start,  pause,  and  restart  the  simulation? 

30.  What  type  of  operator  interaction  is  involved? 

30.1.  Do  you  start  the  model  and  simply  play  back 
the  results? 

30.2.  Is  the  model  interactive  (man-in-the-loop) ? 

31.  Can  you  compare  two  or  more  runs  to  analyze  the 
difference  in  results? 

32.  Is  the  simulation  graphical  (do  you  see  pictorially 
what's  going  on)? 

33.  What  is  the  annual  operations  and  maintenance  cost  for 
the  model  (dollars  and  people) ? 

QUESTIONS  REGARDING  DOCUMENTATION  AND  CONFIGURATION 
MANAGEMENT  OF  EACH  MODEL/SIMULATION 

34.  What  type  of  documentation  exists? 

34.1.  User  guides? 

34.2.  Programmer's  guide? 

34.3.  MIL-STD-2167A  documentation? 


34.3.1. 

Software  Development  Plan? 

34.3.2. 

Software  Requirements  Document? 

34.3.3. 

System  Segment  Specification? 

34.3.4. 

Interface  Requirements 
Specification? 

34.3.5. 

Software  Design  Document? 

34.3.6. 

Test  Plans,  etc.? 
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35.  Is  there  a  configuration  management  plan? 

35.1.  Who  maintains  configuration  management? 

35.2.  How  do  changes/upgrades  get  documented? 

35.3.  How  often  are  updates  made? 

35.3.1.  Who  decides? 

35.3.2.  What  is  the  update  criteria? 

35.3.3.  How  are  tri-Service  users 
involved? 

35.3.4.  How  is  it  funded? 

35.3.5.  Who  does  the  work? 

35.3.5.1.  How  long  does  it 
take  to  do  the 
update? 

35.3.5.2.  If  contracted,  how 
often  do  the  updates 
get  competed? 


AND 


36.  Was  the  model  ever  validated? 


36.1.  What  definition  of  validation  was  used? 

(This  is  especially  important  to  establish 
that  many  different  views  exist  on  what  is 
and  is  not  validation.  Bottom  Line:  The 
generally  accepted  definition  for  model 
validation  says  that  "Validation  is  the 
process  of  determining  the  degree  to  which  a 
model  is  an  accurate  representation  of  the 
real  world  from  the  perspective  of  the 
intended  use  of  the  model." 


36.2.  How  was  validation  conducted  (by  Component 
or  for  the  overall  system) ? 


APPENDIX  Bt  MODEL/SIMULATION  TECHNICAL  QUESTIONS 

(Continued) 


36.3.  Who  validated  it  (Who  did  the  analyses  to 
determine  the  model  represented  the  real 
world) ? 

36.4.  What  documentation  exists  to  support  a 
validation  decision  (Are  the  analyses 
documented  in  some  fashion) ? 

36.5.  What  validation  criteria  were  used  (What 
were  the  items  used  in  the  analyses;  what 
were  the  measures  of  merit  for  these  items)? 

36.6.  Who  developed  the  validation  criteria? 

36.7.  Who  decided  the  validation  criteria  were 
correct? 

36.8.  Who  signed-off  on  the  overall  validation  for 
this  model? 

36.9.  If  the  model  represents  a  foreign  system, 
who  certified  that  the  information  and  the 
representation  of  that  information 
(algorithms)  were  correct?  Are  they  the 
responsible  DoD  office  for  that  system  (read 
DIA  for  threat  type  systems) ? 

37.  Was  the  model  ever  accredited  for  any  particular 

program  (Did  someone  say  that  the  model  was  appropriate 
to  use  for  a  specific  test  or  project)? 

37.1.  Who  made  the  accreditation? 

37.2.  What  authority  did  they  have  to  make  the 
accreditation? 

37.3.  Is  there  any  documentation  to  support  the 
accreditation  decision? 

37.4.  What  was  the  criteria  used  in  the 
accreditation  process? 

37.5.  How  were  the  criteria  developed? 

37.6.  Who  developed  the  criteria? 

37.7.  Who  decided  the  criteria  were  acceptable? 
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38.  Have  the  results  of  the  model  been  compared  to  other 
models? 

38.1.  Did  they  compare? 

38.1.1.  If  not,  why? 

38.1.2.  What  didn't  compare  favorably? 

38.1.3.  Was  your  model  correct? 

38.1.4.  Was  the  other  model  correct? 

38.1.5.  Were  both  models  incorrect? 

39.  Were  the  results  ever  compared  to  the  real  world? 

40.  Are  the  results  repeatable? 

40.1.  Was  repeatability  tested? 

40.2.  Was  it  re-tested  after  the  model  was  changed 
(updated)? 

40.3.  Who  performed  the  testing? 

40.4.  Is  the  testing  documented? 
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APPENDIX  Ct 

ACRONYMS  AND 
m-on-m 


l-on-l 

Str at . 

C2 

C3I 

Log. /Trans. 

EW 

ECM 

Syst.  Perf. 

Syst.  Effec. 

Surv. 

Vuln. 

Leth. 


MODEL  NAME  VERSUS  FUNCTIONAL  AREA  SAMPLE 
RESULTS 


ABBREVIATIONS  FOR  MODEL  FUNCTIONAL  AREAS 

Scenario  with  arbitrary  number  of  players  or 
items,  such  as  m  "blue"  war  game  players 
against  n  "red"  war  game  players,  with  an 
arbitrary  number  of  communication  nodes  or 
arbitrary  number  of  interacting  weapon 
systems . 

Scenario  with  two  opposing  simulation  players 
or  items.  Most  often  a  friendly  weapon 
system  posed  against  a  threat  weapon  system. 

Strategic  weapon  systems  included  as  part  of 
the  simulation  scenario.  Nuclear, 
biological,  and  chemical  weapon  systems  were 
counted  as  strategic  weapon  systems. 

Command  and  Control. 

Command,  Control,  Communications,  and 
Intelligence. 

Logistics  and/or  Transportation  simulation. 
Any  simulation  concerned  with  the  supply, 
storage,  or  transportation  of  materials  on  a 
large  scale. 

Electronic  Warfare.  Infrared  sensor  systems 
have  been  included  as  electronic  warfare 
systems . 

Electronic  Countermeasures. 

System  Performance  Simulation.  Any 
simulation  that  includes  the  behavior  or 
operational  characteristics  of  a  weapon 
system  or  weapon  systems. 

System  Effectiveness  Simulation.  Any 
simulation  that  models  the  ability  of  a 
weapon  system  or  weapon  systems  to  accomplish 
a  task  or  achieve  a  goal. 

Survivability . 

Vulnerability . 

Lethality. 
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APPENDIX  D:  VERIFICATION.  VALIDATION.  AND  ACCREDITATION  SAMPLE  RESULTS  (Continued) 
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APPENDIX  Pi  VERIFICATION.  VALIDATION.  AND  ACCREDITATION  SAMPLE  RESULTS  (Continued) 
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ON  AND  CONFIGURATION  MANAGEMENT  SAMPLE  RESULTS  (Continued) 


67 


68 


69 
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_ pnrnim^rvhafclon _  Configuration  Management/ 

User  Analyst  Programmer  Configuration  Control 

Manual (s)  Manual(s)  Manual(s)  Plan  Policy  Board 


_ Documentation _  Configuration  Management/ 

User  Analyst  Programmer  _ CftDtlgttrfltiQn  Contaftl 

Manual (s)  Manual (s)  Manual (s)  Plan  Policy  Board 
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Configuration  Management/ 
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_ Documentation _  Configuration  Management/ 

User  Analyst  Programmer  _ Configuration  Control 

Manual(s)  Manual (s)  Manual(s)  Plan  Policy  Board 
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Configuration  Kanagmant/ 
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APPENDIX  Fi  PRIOR  AUDITS  AND  STUDIES  RELATED  TO  MODELING 
AND  SIMULATION 


GAO  Report,  No.  GAO/PEMD-88-3  (OSD  Case  No. _ 7336).., - UDoD 

Simulations:  Improved  Assessment  Procedures  Would  Increase 

the  credibility  of  Results. »  December _ 1987,.  The  GAO 

considered  a  framework  of  14  factors  to  evaluate  a 
simulation's  credibility.  These  factors  were  divided  into 
three  major  areas  of  concern:  theory,  model  design,  and 
input  data;  the  correspondence  between  the  model  and  the 
real  world;  and  documentation  and  reporting  issues. 
Evaluating  three  simulations  by  collecting  and  analyzing 
information  using  these  factors  should  help  identify 
strengths  and  weaknesses  of  a  simulation  and,  therefore, 
attest  to  its  credibility.  The  GAO  concluded  that  no 
guidance  existed  at  the  OSD  level  that  could  be  routinely 
used  throughout  DoD  to  review  the  credibility  of  military 
models.  The  GAO  recommended  that  the  secretary  of  the 
Department  of  Defense  direct  agencies  responsible  for 
managing  three  specific  weapon  systems  to  correct  the 
limitations  GAO  had  identified  during  the  audit,  especially 
limitations  in  validation,  and  adopt  or  develop  and 
implement  guidance  on  producing,  validating,  documenting, 
managing,  maintaining,  using,  and  reporting  simulations  of 
weapon  system  effectiveness. 

Inspector  General.  Department  of  Defense.  Report  No.  8 9 -0.57. i. 
"Waraamina  Activities  in  the  Department  of  Defense,"  March 
1989 .  The  auditors  concluded  that  DoD  Service  schools, 
colleges,  and  training  centers  had  expanded  or  planned  to 
expand  wargaming  capabilities  without  clear  objectives  and 
guidance  on  joint  wargaming  activities.  In  addition,  plans 
to  operate  and  expand  computer  facilities  in  support  of 
wargaming  for  NATO  were  difficult  to  justify  in  view  of 
current  operations  or  future  workload.  The  audit  report 
recommended  that  the  Assistant  Secretary  of  Defense  for 
Force  Management  and  Personnel  participate  in  establishing 
policies  and  procedures  on  wargaming  and  similar  activities; 
the  Joint  Chief  of  Staff  designate  an  office  of  primary 
responsibility  within  the  Joint  Staff  to  establish  policies 
and  procedures  for  joint  wargaming  activities  and  clarify 
the  mission  and  responsibilities  of  DoD  colleges,  schools, 
and  training  centers  participating  in  wargaming  activities ; 
discontinue  operation  and  funding  of  the  Joint  Warfare 
Center  because  its  missions  and  functions  overlapped  and  are 
duplicated  elsewhere;  the  Navy  stop  construction  of 
Sensitive  Compartmented  Information  Facilities  at  Naval 
schools,  colleges,  and  training  centers  when  wargaming 
activities  are  limited  to  education  and  training;  the  Air 
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Force  develop  joint  program  management  plans,  establish 
Memorandums  of  Agreement,  and  implement  operating  procedures 
to  manage  effectively  wargaming  activities  of  the  Air  Force 
Wargaming  Center.  In  addition,  the  audit  report  recommended 
that  the  NATO  command  prepare  an  updated  management  plan 
specifying  objectives  of  wargaming  activities,  operating 
procedures,  and  the  requirement  for  facilities  and  computer 
systems  necessary  to  manage  effectively  wargaming  activities 
there;  and  discontinue  funding  to  expand  its  facilities  at 
the  Warrior  Preparation  Center. 

U.S.  Army  Audit  Agency  Report  No.  SW  90-205,  "The  Army  Model 
Improvement  Program."  April  1990.  The  auditors  found  that 
additional  policies  and  procedures  were  needed  to  ensure 
that  models  were  properly  managed,  justified,  approved,  and 
controlled;  acquisition  of  general  purpose  automatic  data 
processing  equipment  and  software  was  not  adequately 
planned,  justified,  and  approved;  the  Army's  planning, 
programming,  and  budgeting  system  had  not  been  fully 
implemented;  the  use  of  higher  order  programming  languages 
and  configurations  of  Army  models  were  not  adequately 
controlled;  and  Corps  Battle  Simulation  Center  staffing 
requirements  were  frequently  not  evaluated,  justified,  and 
documented.  The  audit  report  recommended  that  the  Army 
establish  policies,  procedures,  and  control  authority  to 
permit  the  Army  Model  Improvement  Program  Management  Office 
to  carry  out  assigned  responsibilities;  develop  procedures 
that  will  result  in  an  overall  consolidated  information 
management  plan;  and  improve  procedures  used  to  plan, 
justify,  and  approve  the  acquisition  of  training 
simulations.  In  addition,  the  audit  report  recommended  that 
an  Army-wide  configuration  control  policy  be  established; 
Army-wide  model  documentation  standards  be  established  that 
require  independent  peer  group  reviews  and  approval  of  model 
documentation;  and  to  develop  a  time-phased  plan  to  accredit 
all  Army  analytical,  functional,  and  training  models  and 
include  in  the  accreditation  process  verification, 
validation,  documentation,  and  configuration  control 
charters . 
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APPENDIX  Gi  SUMMARY  OF  POTENTIAL  BENEFITS  RESULTING  FROM 
AUDIT 


Recommendation 

Reference 


A.l. 


A. 2. 


A. 3. 


A. 4. 


Description  of  Benefit 

Economy  and  Efficiency. 
Reduce  duplication  and 
proliferation  of  models 
and  simulations. 


Internal  Control. 
Assist  in  avoiding 
development  or 
procurement  of 
redundant  models  or 
simulations. 


Internal  Control. 

Reduce  duplication  and 
proliferation  of  models 
and  simulations  in  DoD. 


Economy  and  Efficiency. 
Avoid  unnecessary 
development  and  procurement 
of  modeling  and  simulation 
assets. 


Internal  Control. 

Ensure  that  criteria  are 
established  to  accomplish 
verification,  validation,  and 
accreditation  of  models  and 
simulations. 


Amount  and/or 
Type  of  Benefit 

Funds  Put  to 
Better  Use. 

$803  million  of 
unjustified 
funds  over  the 
6 -year  Future 
Years  Defense 
Plan. 

Contributes  to 
the  cost 
avoidance 
claimed  for 
Recommendation 
A.l. 


Contributes  to 
the  cost 
avoidance 
claimed  for 
Recommendation 
A.l. 

Contributes  to 
the  cost 
avoidance 
claimed  for 
Recommendat ion 
A.l. 


Nonmonetary. 
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APPENDIX  G;  SUMMARY  OF  POTENTIAL  BENEFITS  RESULTING  FROM 
AUDIT  (continued) 


Recommendation 

Reference 


B.2. 


C.l. 


Description  of  Benefit 


Internal  Control. 

Ensure  that  models  and 
simulations  used  for 
purposes  of  requirements 
definition,  weapon  systems 
development,  engineering, 
and  testing  and  evaluation 
have  been  properly  verified, 
validated,  and  accredited. 

Economy  and  Efficiency. 
Reduce  unnecessary 
development  and  procurement 
of  modeling  and  simulation 
assets. 


Economy  and  Efficiency. 
Reduce  duplication 
and  proliferation  by 
ensuring  that  authorized 
users  of  models  and 
simulations  have  proper 
documentation  necessary 
to  enable  this  use  to 
occur . 


Amount  and/or 
Type  of  Benefit 


Undeterminable . 


Contributes  to 
the  cost 
avoidance 
claimed  for 
Recommendation 
A.l. 


Contributes  to 
the  cost 
avoidance 
claimed  for 
Recommendation 
A.l. 
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APPENDIX  Hi  ACTIVITIES  VISITED  OR  CONTACTED 


Office  of  the  Secretary  of  Defense 

Office  of  the  Under  Secretary  of  Defense  for  Acquisition, 
Washington,  DC 

Office  of  the  Director,  Defense  Research  and  Engineering, 
Washington,  DC 

Office  of  the  Assistant  Secretary  of  Defense  (Command, 

Control,  Communications  and  Intelligence) ,  Washington,  DC 
Office  of  the  Assistant  Secretary  of  Defense  (Program 
Analysis  and  Evaluation) ,  Washington,  DC 
Defense  Modeling  and  Simulation  Office,  Alexandria,  VA 

Office  of  the  Chairman  of  the  Joint  Chiefs  of  Staff 

Commander  in  Chief,  U.S.  Transportation  Command,  Scott  Air 
Force  Base,  IL 

The  Joint  Staff,  Office  of  the  Director  for  Operations 
( j— 3 ) ,  Joint  Electronic  Warfare  Center,  San  Antonio,  TX 
The  Joint  Staff,  Office  of  the  Director  for  Logistics  (J-4) , 
Washington,  DC 

The  Joint  Staff,  Office  of  the  Director  for  Operational 
Plans  and  Interoperability  (J-7) ,  Washington,  DC 
The  Joint  Staff,  Office  of  the  Director  for  Force  Structure, 
Resources  and  Assessment  (J-8) ,  Washington,  DC 

Department  of  the  Army 

Office  of  the  Chief  of  Staff  of  the  Army,  Test  and 
Evaluation  Management  Agency,  Washington,  DC 
Office  of  the  Deputy  Under  Secretary  of  the  Army  (Operations 
Research) ,  Washington,  DC 
Army  Aviation  and  Troop  Command,  St.  Louis,  MO 
Army  Communications  and  Electronics  Command, 

Fort  Monmouth,  NJ 

Army  Laboratory  Command,  Atmospheric  Sciences 
Laboratory,  White  Sands  Missile  Range,  NM 
Army  Materiel  Command,  Adelphi,  MD 
Army  Missile  Command,  Huntsville,  AL 
Army  Strategic  Defense  Command,  Huntsville,  AL 
Army  Training  and  Doctrine  Command,  White  Sands  Missile 
Range,  NM 

Army  Corps  of  Engineers,  Waterways  Experiment  Station, 
Vicksburg,  MS 

Army  Concepts  Analysis  Agency,  Bethesda,  MD 
Army  Electronics  Proving  Ground,  Fort  Huachuca,  AZ 
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APPENDIX  H:  ACTIVITIES  VISITED  OR  CONTACTED  (continued) 


Department  of  the  Navy 

Office  of  the  Chief  of  Naval  Operations,  Washington,  DC 
Naval  Air  Warfare  Center,  Weapons  Division,  China  Lake,  CA 
Naval  Air  Warfare  Center,  Weapons  Division,  Point  Mugu,  CA 
Naval  Command,  Control  and  Ocean  Surveillance  Center, 
Command  and  Control  Interoperability  Division, 

San  Diego,  CA 

Naval  Surface  Warfare  Center,  Silver  Spring,  MD 
Naval  Undersea  Warfare  Center,  New  London,  CT 
Naval  War  College,  Newport,  RI 

Marine  Corps  Wargames  Assessment  Center,  Quantico,  VA 
Department  of  the  Air  Force 

Headquarters,  U.S.  Air  Force,  Washington,  DC 
Air  Force  Studies  and  Analysis  Agency,  Washington,  DC 
Air  Force  Electronic  Warfare  Center,  San  Antonio,  TX 
Air  Force  Flight  Test  Center,  Edwards  Air  Force  Base,  CA 
Air  Force  Operational  Test  and  Evaluation  Center,  Kirtland 
Air  Force  Base,  NM 

Air  Force  Wargaming  Center,  Maxwell  Air  Force  Base,  AL 

Rome  Laboratories,  Griffiss  Air  Force  Base,  NY 

Wright  Laboratories,  Wright-Patterson  Air  Force  Base,  OH 

Defense  Agencies 

Defense  Advanced  Research  Projects  Agency,  Arlington,  VA 
Defense  Information  Systems  Agency,  Joint  Interoperability 
and  Engineering  Organization,  Fort  Monmouth,  NJ 
Defense  Information  Systems  Agency,  Joint  Interoperability 
and  Engineering  Organization,  Reston,  VA 
Defense  Intelligence  Agency,  Missile  and  Space  Intelligence 
Center,  Redstone  Arsenal,  AL 
Joint  Aircraft  Survivability/Vulnerability  Information 
Analysis  Center,  Wright-Patterson  Air  Force  Base,  OH 
Strategic  Defense  Initiative  Organization,  Washington,  DC 
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APPENDIX  I:  REPORT  DISTRIBUTION 


Office  of  the  Secretary  of  Defense 

Under  Secretary  of  Defense  for  Acquisition 
Director,  Defense  Research  and  Engineering 
Assistant  Secretary  of  Defense  (Command,  Control, 
Communications  and  Intelligence) 

Assistant  Secretary  of  Defense  (Program  Analysis  and 
Evaluation) 

Defense  Modeling  and  Simulation  Office,  Alexandria,  VA 

Office  of  the  Chairman  of  the  Joint  Chiefs  of  Staff 

Commander  in  Chief,  U.S.  Transportation  Command 

The  Joint  Staff,  Office  of  the  Director  for  Operations  (J-3) 

The  Joint  Staff,  Office  of  the  Director  for  Logistics  (J-4) 

The  Joint  Staff,  Office  of  the  Director  for  Operational 

Plans  and  Interoperability  (J-7) 

The  Joint  Staff,  Office  of  the  Director  for  Force  Structure, 
Resources  and  Assessment  (J-8) 

Department  of  the  Army 

Secretary  of  the  Army 

Inspector  General,  Department  of  the  Army 
Office  of  the  Chief  of  Staff  of  the  Army,  Test  and 
Evaluation  Management  Agency 

Office  of  the  Deputy  Under  Secretary  of  the  Army  (Operations 
Research) 

Army  Aviation  and  Troop  Command 

Army  Communications  and  Electronics  Command 

Army  Laboratory  Command,  Atmospheric  Sciences  Laboratory 

Army  Materiel  Command 

Army  Missile  Command 

Army  Strategic  Defense  Command 

Army  Training  and  Doctrine  Command,  White  Sands 
Missile  Range 

Army  Corps  of  Engineers,  Waterways  Experiment  Station 
Army  Concepts  Analysis  Agency 
Army  Electronics  Proving  Ground 
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APPENDIX  It  REPORT  DISTRIBUTION  (continued) 


Department  of  the  Navy 


Secretary  of  the  Navy 
Commandant  of  the  Marine  Corps 

Assistant  Secretary  of  the  Navy  (Financial  Management) 
Office  of  the  Chief  of  Naval  Operations 
Naval  Air  Warfare  Center,  Weapons  Division,  China  Lake 
Naval  Air  Warfare  Center,  Weapons  Division,  Point  Mugu 
Naval  Command,  Control  and  Ocean  Surveillance  Center, 
Command  and  Control  Interoperability  Division 
Naval  Surface  Warfare  Center 
Naval  Undersea  Warfare  Center 
Naval  War  College 

Marine  Corps  Wargames  Assessment  Center 
Department  of  the  Air  Force 
Secretary  of  the  Air  Force 

Assistant  Secretary  of  the  Air  Force  (Financial  Management 
and  Comptroller) 

Headquarters,  U.S.  Air  Force 
Air  Force  Studies  and  Analysis  Agency 
Air  Force  Electronic  Warfare  Center 
Air  Force  Flight  Test  Center 

Air  Force  Operational  Test  and  Evaluation  Center 
Air  Force  Wargaming  Center 
Rome  Laboratories 
Wright  Laboratories 

Other  DoD  Activities 


Director, 

Director, 

Director, 

Director, 


Defense  Advanced  Research  Projects  Agency 
Defense  Information  Systems  Agency 
Defense  Intelligence  Agency 
Strategic  Defense  Initiative  Organization 
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APPENDIX  T;  REPORT  DISTRIBUTION  (continued) 


Non-Defense  Activities 

Office  of  Management  and  Budget 

U.S.  General  Accounting  Office,  National  Security  and 
International  Affairs  Division,  Technical  Information 
Center 

Chairman  and  Ranking  Minority  Member  of  the  following 
Congressional  Committees  and  Subcommittees: 

Senate  Committee  on  Appropriations 
Senate  Subcommittee  on  Defense,  Committee  on 
Appropriations 

Senate  Committee  on  Armed  Services 
Senate  Committee  on  Governmental  Affairs 

House  Committee  on  Appropriations  ... 

House  Subcommittee  on  Defense,  Committee  on  Appropriations 
House  Committee  on  Armed  Services 
House  Committee  on  Government  Operations 

House  Subcommittee  on  Legislation  and  National  Security, 
Committee  on  Government  Operations 
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Comments  from 
Engineering 
Comments  from 
Comments  from 
Comments  from 


PART  IV  -  MANAGEMENT  COMMENTS 

The  Director  of  Defense  Research  and 

The  Joint  Staff 

the  Department  of  the  Army 

the  Naval  War  College 


Comments  from  the  Director  of  Defense  Research  and  Engineering 


DIRECTOR  OF  DEFENSE  RESEARCH  AND  ENGINEERINO 


WASHINGTON,  DC  t«>01-S»1t 


cm  i  o  1993 


MEMORANDUM  FOR  IBS  DoD  INSFSCTOR  GENERAL 

SUBJECT:  Draft  Audit  Raport  on  Duplication/Proliferation  of 

Noapoit  Systems'  Mod* ling  and  Siaulation  Efforts  Within 
DoD  (Project  Mo.  2AB-001S) 

Th*  DoD  XG  audit  was  a  vary  tiaaly  assessment  of  modeling 
and  siaulation  afforta  vithin  tha  Dapartaant.  I  appreciate  tha 
affort  that  was  required  to  look  at  this  vary  complex  sub j act. 

It  is  important  to  not*  that  savarsl  of  your  findings  mirror 
similar  Issues  which  hav*  baan  aurfacad  by  tha  Exacutiva  Council 
on  Modeling  and  Siaulation.  X  concur  with  tha  racoaaandations  in 
th*  raport  provided  that  th*  corrective  action  currently  underway 
for  each  is  stated.  My  main  concern  is  that  th*  raport  does  not 
fully  acknowledge  th*  Department's  plans  and  efforts  to  rectify 
th*  deficianciaa.  Addressing  these  plans  and  efforts 
sufficiently  in  th*  raport  will  help  aak*  th*  raport  a  useful 
aanageaent  tool  to  assist  in  focusing  th*  issues.  My  assessment 
is  that  th*  actions  w*  hav*  underway  should  whan  coaplata  clear 
up  your  concerns. 

In  June  mi,  th*  USD(A),  recognising  th*  need  for  better 
aanageaent  of  modeling  and  siaulation,  established  th*  Defense 
Modeling  and  Siaulation  Office  (DMSO)  as  a  focal  point  for  policy 
and  integration.  Th*  DMSO  has  been  working  since  that  time  on 
th*  issues  you  identified.  Consequently,  a  major  step  in 
resolving  soda  ling  and  siaulation  issues  will  be  th*  issuance  of 
DoDD  50XX.XX,  -Modeling  and  Siaulation."  In  addition  to  updating 
th*  audit  raport  to  reflect  th*  ongoing  efforts  and  th*  progress 
made  in  eliminating  th*  deficiencies,  I  suggest  that  the  Office 
of  th*  Inspector  General  endorse  and  recoaaend  th*  expeditious 
issuance  of  DoDD  50XX.XX,  -Modeling  and  Simulation.* 

Comments  on  th*  audit  raport  racoaaandations  as  prescribed 
by  DoD  Directive  7*50.3,  paragraphs  P.b(l)  and  F.b(2),  are 
provided  at  Attachment  1.  Additional  detailed  comments  on  th* 
draft  report  are  at  Attachment  2.  These  detailed  comments 
provide  additional  information  on  several  items  and  also  correct 
a  number  of  technical  errors.  Their  inclusion  will  improve  the 
overall  accuracy  and  completeness  of  th*  final  report.  If  v*  can 
be  of  any  further  assistance  in  your  effort,  ay  point  of  contact 
for  DMSO  is  Col.  Ed  Pitssiaaons,  (703)  990-0**0. 


Attachments 
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Comments  from  the  Director  of  Defense  Research  and  Engineering  (continued) 


htTACMMOT  i 


Director,  Mtuii  knureb  u4  Engineering  Ccnuti  m  Mit 

iMoantetlou 


A*  proscribed  In  DoD  Directive  7*50.3,  Section  F.,  paragraphs 
b(l)  end  b(2) ,  the  following  consents  ere  in  response  to  the 
reconnendations  nade  in  the  DoD  Inspector  General's  "Draft  Audit 
Report  on  Duplication/Proliferation  of  Weapons  Systeas'  Modeling 
and  simulation  If  forts  within  DoD  (Project  Wo.  2AB-0016),  October 
33,  1992."  All  reconnendations  were  proposed  for  Under  Secretary 
of  Defense  for  Acquisition  (USD(A))  action. 


part  it. a.  Duplication.  Redundancy,  end  Proliferation -(Pago  29) 


1.  Assign  responsibility  for  development  of  policies  and 
procedures  related  to  investnent,  internal  developnent, 
interoperability  standards,  nodification  of  existing  assets,  and 
naintenance  of  catalogues  regarding  sods  ling  and  simulation 
activities  within  DoD.  Amy  regulation  5-11  could  provide  the 
baseline  for  developnent  of  these  DoD  procedures. 

Response;  Concur  provided  that  the  recoanendation  in  the  report 
include  a  statement  that  corrective  action  is  in  process. 
Corrective  action  taken  stems  fron  the  21  June,  1991  ,  Deputy 
Secretary  of  Defense  senorandum  and  USD  (A)  memorandum  of  2* 
September  1991.  These  memorandum  named  the  members  of  the  DoD 
executive  Council  for  Models  and  Simulations  (EXCIMS)  and 
designated  the  Director,  Defense  Research  and  Engineering  as  the 
chair.  The  draft  DoDD  50 XX. XX,  "Modeling  and  Sinulation"  (herein 
referred  to  as  DoDD  50XX.XX),  currently  in  coordination,  will 
continue  the  directions  outlined  in  the  memorandum  and  establish 
policies.  DoDD  50XX.XX  states  that  the  USD(A)  shall:  (1)  be 
responsible  for  strengthening  the  use  of  modeling  and  simulation 
within  DoD,  and  (2)  issue  plans,  policies,  directives,  procedures, 
and  publications  for  management  and  advancement  of  modeling  and 
sinulation. 


3.  Develop  a  standard  reporting  procedure  for  starting  new 
modeling  and  sinulation  efforts  with  a  cost  threshold  exceeding 
$500,000  for  input  into  a  DoD-wide  catalogue  system  and  require 
all  Components  to  comply  with  this  procedure. 

Response:  Concur  provided  that  the  recommendation  in  the  report 
include  a  statement  that  corrective  action  is  in  process.  In 
addition  to  the  efforts  underway  or  planned  due  to  DoDD  50XX.XX, 


Comments  from  the  Director  of  Defense  Research  and  Engineering  (continued) 


the  Defense  Modeling  and  » illation  Of rice  is  developing  a 
prototypa  electronic  bullatin  board  and  raposltory  with 
information  clearinghouse  capabilities  such  as  cataloguas. 
announcaaants ,  lists  of  significant  events,  and  documents.  Poll 
oparational  capability  is  projected  by  June,  »M.  A  major 
component  of  the  system  will  be  a  directory  MS  cataloguas. 


3.  Require  that  DoO  Components  establish  formal  oversight 
responsibilities  and  associated  internal  controls  for  modeling  and 
simulation  activities,  based  on  Army  Regulation  5-11,  "Army  Nodal 
and  Simulation  Management  Program." 


Response;  Concur  provided  that  the  recommendation  in  the  report 
include  a  statement  that  corrective  action  is  in  process.  The 
intent  of  this  recommendation  was  captured  in  the  policy  section 
of  DoDD  50XX.XX,  which  states  that  "DoD  component  management 
systems  for  modeling  and  simulation  oversight"  are  required. 


4.  Establish  and  maintain  a  common  DoD  modeling  and  simulation 
library  with  ready  access  to  modeling  and  simulation  users  and 
developers. 

Response;  Concur  provided  that  the  recommendation  in  the  report 
include  a  statement  that  corrective  action  is  in  process.  DoDD 
50 XX. XX  allows  for  the  establishment  of  a  common  library. 
Specifically,  the  directive  requires  the  development  of  common 
tools,  methodologies, and  databases  in  modeling  and  simulation; 
and,  establishment  of  standards  and  protocols.  The  directive  also 
requires  the  creation  of  a  modeling  and  simulation  information 
and  data  center. 


PART  II. b.  Verification.  Validation,  and  Accreditation  fWIAl 
IPaq*  4il 


1.  Using  the  Army  Model  and  Simulation  Management  Program  as  a 
framework,  develop  policy,  guidance,  standards,  and  criteria  by 
which  verification,  validation,  and  accreditation  of  models  and 
simulations  is  to  be  accomplished. 

Response;  Concur  provided  that  the  recommendation  in  the  report 
Include  a  statement  that  corrective  action  is  in  process.  DoDD 
50XX.XX  establishes  requirements  for  verification,  validation,  and 
accreditation. 


3.  Require  that  models  and  simulations  used  for  requirements 
definition,  weapon  system  development,  engineering,  and  testing 
and  evaluation  be  verified,,  validated,  and  accredited  in 
accordance  with  policy,  guidance,  standards,  and  criteria  of 
Rftcoutndation  1. 


Comments  from  the  Director  of  Defense  Research  and  Engineering  (continued) 


Comments  from  the  Director  of  Defense  Research  and  Engineering  (continued) 


imcmn  t 

Director,  Mua«  iMMtoh  ul  Engineering  MUtM  Cmuti 

The  following  comments  are  in  response  to  the  DoO  Inspector 
General's  'Draft  Audit  Report  on  Duplication/Prol  if  oration  of 
Meapons  Systems'  Modeling  and  Simulation  Efforts  within  DoD 
(Project  Mo.  2AB-0016) ,  Octobar  32,  1992. 


GENERAL 

The  Director,  Dafonso  Research  and  Engineering  (DORSE), 
through  the  Defense  Modeling  and  Siaulation  Office  (DMSO),  is  in 
the  process  of  iapleaenting  aany  of  the  recommendations  stated  in 
this  draft  audit.  Because  the  report  does  not  adequately  address 
the  plane  and  efforts  already  underway,  the  report  is  somewhat 
misleading  and  detracts  from  the  important  issues  that  are 
addressed.  If  a  section  was  included  in  the  report  that  addressed 
the  current  status  and  anticipated  impact  of  the  plans  and  efforts 
underway  we  feel  the  report  would  then  become  a  useful  management 
tool  to  assist  in  completing  the  corrective  actions  you  have 
addressed. 

Detailed  comments  on  the  Executive  Summary  (Pages  1  and  ii) 
are  not  provided  here.  It  is  assumed  that  the  Executive  Summary 
will  change  to  reflect  any  comments  incorporated  in  the  body  of 
the  report. 

The  comments  that  follow  correspond  to  the  sections,  page 
numbers  and  paragraphs  of  the  draft  report. 


PART  I  -  INTRODUCTION 

Background  if ago*  1-11 

Paragraph  1:  Recommend  that  the  opening  paragraph  utilise  the 
definitions  that  have  been  developed  for  modeling  end  simulation 
in  the  draft  DoD  Directive  DoDD  50XX.XX,  •Modeling  and  Simulation" 
(herein  referred  to  as  DoDD  SOXX.XX) .  This  would  eliminate 
potential  confusion  to  readers  already  familiar  with  the  defense 
modeling  and  simulation  program.  (Modal?  A  physical, 
mathematical,  or  otherwise  logical  representation  of  a  system, 
entity,  phenomenon,  or  process,  simulation?  A  method  for 
iapleaenting  a  model  over  time,  as  well  as  a  technique  for 
testing,  analysis,  or  training  in  which  real  world  and  conceptual 
systems  are  reproduced  by  the 

Paragraph  3:  Recommend  the  DoD-approved  definition  of 
"operations  research”  be  used  in  this  report  (JCS  Pub  1-02,  The 
Department  of  Defense  Dictionary  of  Military  and  Associated 
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Comments  from  the  Director  of  Defense  Research  and  Engineering  (continued) 


Tiru) .  The  remainder  of  the  report  oppoaro  to  use  ■operetlone 
research"  as  a  codeword  for  modeling  and  simulation.  Using  the 
standard  definition  for  operetiono  research  it  is  not  a  codeword 
for  modeling  and  simulation  and  we  therefore  recommend  that  the 
abbreviation  "MSS"  replace  "operations  research"  throughout  the 
document. 

Paragraph  St  Recommend  add  "unnecessary"  before  duplication. 
Change  "...for  modeling  in  DoO..."  to  include  "for  strengthening 
the  use  of  modeling  and  simulation  in  DoD..."  This  sore  accurately 
reflects  the  direction  of  the  Deputy  Secretary  of  Defense 
Memorandum,  "Modeling  and  Simulation  Management  Plan",  21  June, 
1991,  and  the  responsibilities  assigned  to  USD(A)  in  DODD  50XX.XX. 

flbligiiauL  iEisat  JJ. 

Paragraph  1:  The  term  "unnecessary  investment  in  modeling"  in 
the  second  sentence  is  not  defined.  "Necessary"  and  "unnecessary" 
modeling  should  be  defined  in  this  section  so  that  the  report's 
objectives  are  better  understood. 

Sssssl  fPagti  2=51 

Paragraph  1:  The  method  of  statistical  sampling  should  be 
explained  and  data  provided. 

Paragraph  4:  This  paragraph  states  that  the  audit  scope  was 
restricted  to  operations  research  assets.  The  conclusions  are 
then  generalised  to  all  of  MSS.  There  is  no  cogent  argument 
offered  to  show  that  operations  research  models  accurately  reflect 
training  and  other  models. 

Other  Matters  of  Interest  7-1Q1 

Paragraphs  1-3:  Recommend  these  paragraphs  be  omitted  from 
this  report  as  they  are  not  germane  to  weapons  systems'  MiS.  If 
retained,  recommend  that  "MAS"  be  substituted  for  "operations 
research'  throughout  the  section.  These  studies  dealt  with 
specific  uses  of  modeling  and  simulation,  not  operations  research. 
The  statement  "All  three  studies  concluded  there  were  managerial 
and  technical  difficulties  in  implementing  operations  research  in 
the  DoD"  cannot  be  derived  from  the  Individual  studies. 

Paragraph  4:  Recommend  reword  to  reflect  final  version  of 
report,  "A  Review  of  Study  Panel  Recommendations  for  Defense 
Modeling  and  Simulation",  June  1992.  Add  "and  completed  for  the 
DMSO"  after  "1992".  The  report  should  also  mention  that  the  IDA 
report  recommendatione  provide  the  foundation  for  the  work  of  DMSO 
and  that  each  recommendation  was  categorised  with  regard  to  which 
DMSO  general  objective,  application  area,  and  technical  objective 
it  addressed. 
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Comments  from  the  Director  of  Defense  Research  and  Engineering  (continued) 


PART  XX  -  FINDINGS  AND  RECOMMENDATIONS 

A.  DUPLICATION,  REDUNDANCY,  AND  PROLIFERATION  (Pag*  11) 

Paragraph  It  The  rtcoiMndttiom  of  Finding  A  atata  that  tha 
Undar  Sacratary  of  Dafanaa  of  Acquisition  (USD (A))  should  assign 
rasponsibility  for  development  of  pollcias  and  procaduras  for 
sodaling  and  sisulation.  This  sactlon  should  ba  updatad  to 
raflact  tha  Department ' a  af forts.  Tha  21  Juno,  1991,  Daputy 
Sacratary  of  Dafans*  aaaorandua  and  tha  USD(A)  memorandum  of  2S 
Saptaabar  1991  naaad  tha  naabars  of  tha  DoD  Executive  Council  for 
Nodals  and  siaulations  (KXCIMS)  and  designated  tha  Director, 
Dafans*  Research  and  Engineering  (DORSE)  as  the  chair. 

The  draft  DOW)  50XX.XX,  currently  in  coordination,  will 
continue  the  directions  outlined  in  the  aaaorandua  and  establish 
policies.  DODD  50XX.XX  states  that  tha  USD(A)  shall  (1)  ba 
responsible  for  strengthening  tha  us*  of  sodaling  and  sisulation 
within  DoD,  and  (2)  issue  plans,  policies,  directives,  procaduras, 
and  publications  for  aanagaaant  and  advanceaent  of  sodaling  and 
sisulation.  In  tha  "Policy"  section,  raquiraaants  are  stated  for: 
(1)  a  DoD  sodaling  and  sisulation  aanagaaant  and  adainistrativ* 
structure,  (2)  a  sodaling  and  sisulation  aastar  plan,  (2)  a 
sodaling  and  sisulation  invastaant  plan,  (4)  davalopaant  of  coason 
tools,  aetbodologias,  and  databases  in  sodaling  and  sisulation; 
and,  astablishaant  of  standards  and  protocols,  (5)  a  sodaling  and 
sisulation  information  and  data  canter,  («)  general  use 
applications  in  sodaling  and  sisulation,  (7)  varification, 
validation,  and  accreditation  for  sodaling  and  sisulation 
applications,  (•)  sodaling  and  sisulation  applications  to  support 
DoD  acquisition  and  oversight  decision  Baking,  (9)  DoD  component 
sanagesent  systems  for  sodaling  and  sisulation  oversight,  (10)  DoD 
coaponent  representation  in  sodaling  and  sisulation  applications, 
and  (11)  data  and  data  administration  for  sodaling  and  simulation 
applications  to  conform  to  established  policies  and  procedures  for 
data  administration. 


DISCUSSION  OF  DETAILS 

DoD  Management  of  Operations  Research  Activities  (Pages  13-151 

Paragraph  2:  Add  tha  following  statement  to  tha  and  of  tha 
paragraph:  "Two  positive  steps  taken  towards  rectifying  tha 
deficiencies  noted  were  tha  issuance  of  the  Deputy  Secretary  of 
Defense  semorandus  dated  June  21,  1991,  Modeling  and  Simulation 
Management  Plan  and  the  Under  Secretary  of  Defense  (Acquisition) 
memorandum  dated  July  22,  1991,  Establishment  of  tha  Defense 
Modeling  and  Sisulation  Office  (DKSO) . 

Paragraph  5:  Change  "Intarpretability"  in  the  last  line  to 
"  Xnteroperabi  1  i  ty  ” . 


■SOB 


Comments  from  the  Director  of  Defense  Research  and  Engineering  (continued) 


Mod* ling  and  simulation  Libraries  1S-M1 

Paragraph  It  Agree  that  a  master  reference  list  should  ba 
established  that  will  allow  users  to  sort  or  create  catalogues  a* 
needed,  hovavar  tha  OMSO  N48  InformationSystem  should  be 
mentioned.  OMSO  MAS  Information  System  is  a  prototype  electronic 
bulletin  board  and  repository  with  information  clearinghouse 
capabilities  such  as  catalogues,  announcements,  lists  of 
significant  events,  and  documents.  Full  operational  capability  is 
projected  by  June,  1593.  A  major  component  of  the  system  will  be 
a  directory  of  MSS  catalogues. 

Potential  Monetary  Benefits  (Pages  2fc221 

Paragraph  2:  Suggest  you  verify  the  $803*  savings  estimated 
by  your  technical  assistant.  The  number  appears  high  and  may  not 
include  the  cost  of  WAA,  which  should  have  been  included.  If 
this  is  the  case  the  number  should  be  corrected  throughout  the 
report.  Also,  change  "redundancy"  to  "duplication". 

Conclusions .  (Pages  H=3A1 

Paragraph  2:  Agree  with  the  finding  regarding  absence  of 
standards.  However,  w*  recommend  that  the  following  statement  be 
added  to  reflect  ongoing  activity  in  this  arsa:  "Thar*  is 
currently  a  DMSO/DARPA  effort  to  develop  and  implement  the 
Aggregate  Level  System  Protocol  (ALSP)  which  will  allow  service 
developed  wargames  to  communicate  and  interoperate."  This 
capability  has  been  demonstrated  successfully  in  two  major 
exercises. 


B.  VERIFICATION,  VALIDATION,  AND  ACCREDITATION  (Pag*  31) 


Conclusion*  l Pages  42-44) 

Paragraph  2:  Recommend  deleting  the  last  sentence.  It  would 
be  very  costly  to  prescribe  an  enforceable  single  WAA  standard. 
The  WAA  process  needs  to  be  defined  by  the  component  as 
prescribed  in  DODO  50XX.XX.  In  many  cases,  the  cost  of  WAA  in 
time,  personnel,  and  money  could  not  be  justified  compared  to  the 
expected  return  on  investment. 

Paragraph  4:  DODD  50XX.XX  will  apply  to  all  DoD  components. 


C.  CONFIGURATION  MANAGEMENT  AND  DOCUMENTATION  (Pag*  45) 


Recommend  adding  a  discussion  that  addresses  the  existing 
efforts  in  modeling  and  simulation  standards  that  contribute  to 
standardisation  and  configuration  management.  These  Include t  the 
Aggregate  Level  Simulation  Protocol  (ALSP) ,  the  Distributed 
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Comments  from  the  Director  of  Defense  Research  and  Engineering  (continued) 


Interactive  Simulation  (DIE) ,  and  tha  Dafanaa  Slaulatlan  Internet 
(OSX) . 

Work  on  ALSP  was  begun  in  1990  by  tha  Dafanaa  Advanced 
Research  Pro j acta  Agency  (DARPA)  and  la  currant ly  chart ar ad  by 
OSD(A)  through  tha  KXCHtS.  Tha  purpose  of  ALSP  la  to  daalgn  an 
intarfaca  for  aultipla  combat  simulations.  ALSP  dafinaa  tha 
procaaa  for  DoD  to  connact  divaraa  existing  aiaulationa  and  parait 
tha  gaographical  diatributlon  of  war  gaaa  partielpanta.  Within 
ALSP,  an  Xntarfaca  Working  Croup  davalopa  fluid  intarfaca 
protocola,  coordinates  Sarvico  and  Joint  Agancy  alaulation 
anhancaaanta,  and  providaa  ayataa  laval  software.  ALSP  aupporta 
an  intagratad  multi-function  training  anvironaant  for  Joint  and 
coabinad  axarciaaa  through  configuration  managanant  of  ALSP 
protocols  and  ayataa  aoftvara,  incorporation  of  nav  functionality, 
and  daploynant  of  toola,  docuaantatlon,  and  ayataa  aoftvara.  ALSP 
haa  baan  uaad  to  link  tha  Amy's  Corpa  Battla  Siaulation  (CBS) 
with  tha  Air  Force's  Air  War  Simulation  (AWSIM) .  Tha  Wavy  and 
Marina  Corps  hava  also  agraad  to  usa  ALSP  as  tha  intarfaca  batvaan 
the  Wavy's  nav  version  of  tha  Enhanced  Naval  Warfare  Casing  System 
(ENWGS)  and  tha  Marina  Corps's  Marina  Air-Cround  Task  Pores 
(KAGTF)  Tactical  Warfare  Simulation  (MTWS) . 

Another  DARPA  effort,  under  DMSO  sponsorship,  is  tha  DIS. 

Whan  fully  developed,  DXS  will  allow  dissimilar  simulators 
distributed  over  a  large  geographical  area  to  interact  in  a  team 
anvironaant.  Tha  simulators  coamunicata  ovar  local  and  vida  araa 
natvorks.  Since  19S9,  workshops,  involving  industry,  academia, 
and  government  hava  bean  conducted  at  tha  University  of  Central 
Florida's  Institute  for  Simulation  and  Training  to  develop  a 
common  standard  for  describing  tha  fora  and  types  of  messages  to 
be  exchanged  between  siaulated  entities  in  a  DIS.  A  final  draft 
standard  was  submitted  to  tha  Institute  of  Electrical  and 
Electronic  Engineers  (IEEE)  and  has  bean  issued  as  IEEE  Draft 
Standard  P127S. 

Finally,  with  support  froa  DMSO,  DARPA  and  tha  Defense 
Information  Systems  Agency  (DISA)  are  developing  DSX.  DSX  is  a 
high  capacity  network  and  supports  a  full  spectrum  of  war  fighting 
siaulation  interoperability  activities.  D8X  is  intended  to  be 
transitioned  into  a  core  component  of  DISA's  Defense  Information 
Systems  Metvork  to  ensure  that  DoD's  next  generation  corporate 
network  will  meat  tha  needs  of  DIS.  DSX  is  a  high  performance, 
vida  araa,  packet  switched  network  and  supports  tha  infrastructure 
for  a  DoD  seamless  varfighting  simulation. 

Tha  standardisation  and  configuration  management  controls 
embodied  by  tha  ALSP,  DXS,  and  DSX  will  enable  tha  communications 
and  computing  elements  of  DoD  components  to  function  as  a  seamless 
simulation  system,  and  place  tha  power  of  simulation  at  tha 
components'  disposal  for  training,  acquisition,  and  mission 
support  at  costs  significantly  lass  than  that  required  for 
separate  simulations. 
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Comments  from  the  Director  of  Defense  Research  and  Engineering  (continued) 


DISCUSSION  OF  DKTAIU 


BiggiaiaflaUani  tan  corrective  Act  l  on  . /r tot  .12) 

Chang*  tha  opanlng  aantanca  to  "Corrective  action  for  tha 
deficienciea  noted  in  thia  report  can  ba  accoapliahad  with  tha 
iaauanca  of  DoDO  50XX.XX,  thia  villi* 


98 


Comments  from  The  Joint  Staff 


IMi  JOMT  STAMP 


Reply  ZIP  Code: 
20318*8000 


J-8A  00064*83 
22  January  1993 


MEMORANDUM  FOR  THE  DEPARTMENT  OP  DEFENSE  INSPECTOR  GENERAL 
Attention:  Director  for  Acquisition  Management 

Subject:  Audit  on  Duplication/Proliferation  of  Weapon  Systems' 
Modeling  and  Simulation  Efforts  Within  DOD  (Project  Mo 
2AB-0016) 

1.  The  subject  Audit  provides  an  excellent  assessment  of  the 
status  of  DOD  policy  and  guidance  in  the  areas  of  Modeling  and 
Simulation  (M£>S)  standards  and  Verification,  Validation,  and 
Accreditation  (WRA)  to  date.  Our  review  of  the  Audit 
disclosed  three  points  for  which  we  offer  clarification  and 
request  correction  in  the  Audit's  final  report. 

2.  Page  2.  last  sentence.  Change  to:  ‘This  initiative  is  to 


structure.  Resources,  and  Assessment,  director  ate.  J8.  to  con¬ 
solidate  the  modeling  and  sianilation  plans  of  the  commanders  in 
chief,  cut  costs,  and  reduce  duplication." 

REASON:  Accuracy.  The  draft  Audit  confuses  the  purpose 
and  responsibilities  of  the  new  Joint  Sianilation  and  Inter¬ 
operability  Division  ( JSXD) ,  J*7,  with  the  responsibilities 
of  the  Force  Structure,  Resources,  end  Assessment 
Directorate,  J-8.  Responsibilities  for  management  of  joint 
MRS  including  joint  MRS  planning  and  resource  allocation 
are  assigned  on  the  Joint  Staff  to  the  J-8.  Copies  of  the 
pertinent  J-7  and  J-S  mission  stateswnts  from  Joint  Admin 
Pub  1.1,  "Organisations  and  Functions  of  the  Joint  Staff," 
are  enclosed.  JSID  waa  established  by  J-7  to  "promote  the 
application- of  MRS*  and  to  assist  the  J*8's  management  of 
joint  MRS  by  providing  an  operational  perspective  on  the 
need  for  new  MRS  capabilities.  The  DOD  IG  Audit  is 
directed  primarily  at  joint  MRS  management  functions  that 
are  managed  by  the  J-t. 


Comments  from  The  Joint  Staff  (continued) 


3.  Paaa  IS.  la*  mn«nli.  lot  two  lines.  Delete,  ”,  would 
establish  a  Joint  Simulation  and  Interoperability  Division.” 

reason;  Accuracy.  J-7/JSID  is  already  astablishod  and  its 
■istions  and  (unctions  codified  in  Joint  Ad* in  Pub,  1.1, 
'Organisations  and  Functions  of  the  Joint  Staff.”  The 
proposed  MOP  €8  does  not  change  the  organisation  of  the 
Joint  Staff. 


Change  the  ”0”  ratings  to  "N/A.” 

season :  Clarity.  The  JAWS  and  BUILDUP  aodels  were 
cancelled  in  1989  and  1980  respectively.  The  JAWS  aodel  was 
never  used.  A  aore  accurate  assessment  of  the  status  of 
both  models  would  be  'Hot  Applicable.”  A  rating  of  *0,” 
meaning  'None,”  conveys  the  inpression  of  a  deficiency, 
since  the  *0”  rating  was  also  used  to  grade  operational 
models  that  were  deficient  in  Accreditation  and 
Configuration  Management. 

S.  The  corrections  offered  may  help  the  Audit  convey  a  aore 
accurate  picture  and  help  reduce  confusion  surrounding  Joint 
Staff  management  of  MS .  CDR  Robert  G.  Brewer,  USB,  (703) 
614-7881,  is  my  designated  point  of  contact  for  this  action. 
Please  contact  bia  if  any  further  assistance  is  required. 


CONRAD  <T.  LAUTEMBACHKR,  JR. 
Rear  Admiral,  USB 
Director  for  Force  Structure, 
Resources,  and  Assessment 


Bnclosure 


Comments  from  the  Department  of  the  Army 


MBS -OR 


DEPARTMENT  or  TME  ARMY 
OPPICI  OP  TME  UNDER  SECRETARY 
WASMINETOM.  R  C.  MIIMIU 


December  31,  1392 


MEMORANDUM  FOR  IS,  DOD  (AUDITUW) 


K/gXski 


SUBJECT:  Audit  Report  on  Duplication/ Proliferation  of  Weapon 

Systeae  Modeling  and  Siaulation  Efforts  Within  DoD 
(Project  Ho.  2AB-0016) 


Reference  is  aade  to  your  aeaorandua  of  23  October  1992, 
subject  as  above,  which  requested  review  and  coaaant  on  subject 
draft  audit  report. 


The  Aray  fundaaentally  agrees  with  the  general  thrust  of  the 
audit  report  and  is  pleased  to  see  that  our  efforts  to  iaprove  the 
aanageaent  of  aodeling  and  eiaulation  have  been  assessed  as  being 
acceptable  as  a  model  for  other  sectors  of  DoD.  In  our  judgement, 
however,  the  report  in  its  current  fora  contains  substantive  errors 
and  aisuses  of  terminology  which  materially  detract  from  its 
overall  credibility.  We  recoaaend  that  the  report  be  rewritten  and 
then  restaffed  for  consents  prior  to  being  issued.  There  are  three 
principal  reasons  for  this: 


i.  The  report  does  not  uncover  any  issues  that  are  new  to  the 
modeling  and  simulation  community.  The  coordination  of  aodeling 
and  simulation  efforts;  the  validation,  verification, 
accreditation,  and  configuration  control  of  models  and  simulations; 
and  the  adequate  documentation  of  models  and  simulations  are  all 
difficult  problems  that  have  been  with  us  for  a  long  time.  The 
background  presented  and  conclusions  reached,  in  each  of  the  three 
major  areas  discussed,  indicate  an  incomplete  understanding  ef  the 
issues  involved.  For  example,  the  definitions  of  validation  and 
verification  and  the  methods  used  to  apply  them  to  models  and 
simulations  are  much  debated  topics  within  the  community.  The  fact 
that  there  is  no  DoD  mandate  to  perform  WfcA  is  not  the  reason  that 
developers  do  not  seem  to  do  an  adequate  job  in  this  area.  Quite 
the  contrary,  there  is  much  serious  professional  discussion  and 
debate  about  precisely  what  constitutes  adequate  Wtt  and  hew  and 
when  to  apply  the  various  techniques  that  are  available.  As  it  is 
presently  cast,  the  report  does  not  recognise  the  ongoing  nature 
of  the  activities  which  are  making  incremental  edvanoes  toward 
resolving  these  and  other  essentially  technical  issues. 
Bureaucratic  fiat  will  not  resolve  the  Issues  involved. 


b.  The  baseline  sample  used  for  the  audit  was  extracted  from 
a  document  that  is  not  reliable,  complete,  or  current.  In 
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Comments  from  the  Department  of  the  Army  (continued) 


MWOR  Ow«b«r  «,  1*M 

SUBEJCTi  Audit  Report  on  Duplication/Rroliferation  of  Weapons 
Systems  Model  in?  and  Simulation  Efforts  Within  OoO 
(Project  Mo.  2AB-001S) 

addition,  models  not  used  in  the  weapons  systems  acquisition 
process  vers  intermingled  with  and  Judged  against  the  mm  criteria 
as  those  used  in  the  weapons  systen  acquisition  process.  Models 
and  simulations  osed  in  the  product  development  stages  of  the 
acquisition  process  normally  do  not  include  representations  of  the 
entire  combined  anas  team  at  item  systea  level  because  of  tbs 
differing  nature  of  the  problems  and  alternatives  being  analysed. 
However,  those  models  and  simulations  used  for  Cost  and  Operational 
Effectiveness  Analyses  (COBA)  frequently  Include  both  item  system 
and  aggregate  level  representations  of  forces  in  the  combined  arms 
team.  The  best  current  Army  source  of  these  ooabat  effectiveness 
models  is  the  TRADOC  Model  Catalog,  which  is  formally  revised  and 
updated  on  a  two-year  cycle.  In  addition  the  Army's  FORCEM  and 
TRANSMO  models  are  not  used  for  weapons  systems  analysis  at  any 
level,  but  are  used  primarily  in  the  total  force  programming  arena. 

c.  Throughout,  the  audit  report  inappropriately  attempts  to 
use  the  term  "operations  research*  as  a  synonym  for  "modeling  and 
simulation*.  In  many  instances  the  interpretation  of  the  scope  of 
the  audit  is  significantly  altered  by  this  incorrect  substitution. 
Models  and  simulations  are  merely  one  set  of  tools  used  by 
operations  research  practitioners,  and  it  is  incorrect  to  infer 
that  their  use  defines  what  operations  research  is  all  about.  The 
detailed  Army  comments,  which  are  attached  at  Enclosure  1,  include 
sections  which  attempt  to  correct  most  of  these  errors.  After  they 
are  applied,  however,  there  may  be  other  sections  of  the  report 
which  will  need  to  be  rewritten  to  ensure  that  the  proper  context 
of  the  audit  is  maintained.  Therefore,  we  recommend  that  the 
report  be  rewritten  and  then  restaffed  in  its  modified  form. 

The  findings  and  recommendations  are  not  directly  addressed 
to  the  Army.  We  generally  concur  with  the  recommendations  insofar 
as  they  address  the  need  for  integrated  policies  at  the  DoD  level, 
we  do  not  agree  that  ail  of  the  speoifio  findings  are  justified  for 
the  reasons  stated  above,  the  details  of  which  are  found  in  the 
specific  oomments  at  Enclosure  1. 

My  IOC  is  COL  Gilbert  Brauch,  Chief,  Army  Model  and  Simulation 
Management  Office,  (703)«07-337S,  PAR  (703)«07-33Sl,  OWN  327-xxxx. 


Walter  W.  Hollis 

Deputy  Under  Secretary  of  the  Army 
(Operations  Research) 


Enclosure 

as 
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